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Overview 


Opportunity NYC-Family Rewards, an experimental, privately funded, conditional cash trans- 
fer (CCT) program to help families break the cycle of poverty, was the first comprehensive 
CCT program in a developed country. Launched in 2007 by New York City’s Center for Eco- 
nomic Opportunity (CEO), Family Rewards offered cash assistance to low-income families to 
reduce immediate hardship, but conditioned that assistance on families’ efforts to build up their 
“human capital” to reduce the risk of longer-term and second-generation poverty. The program 
thus tied cash rewards to pre-specified activities and outcomes in children’s education, families’ 
preventive health care, and parents’ employment. It operated as a pilot program for three years, 
concluding, as planned, in August 2010. 

Six community-based organizations, in partnership with a lead nonprofit agency, ran 
Family Rewards in six of New York City’s highest-poverty communities. MDRC is evaluating 
the program through a randomized control trial involving approximately 4,800 families and 
1 1 ,000 children, half of whom could receive the cash rewards if they met the required conditions, 
and half who were assigned to a control group that could not receive the rewards. This report 
presents final results on the experience of operating the program and interim findings on its ef- 
fects on a wide range of outcomes three to four years after participants entered the program. Fu- 
ture reports will present longer-term findings. 


Key Findings 

Family Rewards transferred over $8,700, on average, to families during the three-year period. 

As of spring 2013, it had had some positive effects on some outcomes, but left other outcomes 

unchanged. For example, the program: 

• Reduced current poverty and material hardship, including hunger and some housing-related 
hardships, although those effects weakened after the cash transfers ended 

• Helped parents increase savings and reduce reliance on families and friends for cash loans 

• Did not improve school outcomes overall for elementary or middle school students, perhaps 
in part because, for these children, the program rewarded attendance (which was already 
high) and standardized test scores (rather than more immediate performance such as good 
report card grades) 

• Had few effects on school outcomes for high school students overall, but substantially in- 
creased graduation rates and other outcomes for students who entered high school as profi- 
cient readers 

• Did not increase families’ use of preventive medical care, which was already high, and had 
few effects on health outcomes 

• Substantially increased families’ receipt of preventive dental care 

• Increased the likelihood of self-reported full-time employment but did not increase em- 
ployment in or earnings from jobs covered by the unemployment insurance system. 

Building on the early evidence that is emerging from this evaluation, MDRC and CEO 

have revised the Family Rewards model considerably, and MDRC is now testing that new ver- 
sion in Memphis, Tennessee, and the Bronx, New York, in a separate demonstration project. 
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Preface 


The struggle to find effective ways to help low-income populations escape poverty without in- 
creasing long-term and multigenerational reliance on government has been with us for many 
years. Conditional cash transfer (CCT) programs represent one approach that has met with some 
success in lower- and middle-income countries. But until 2007, when Opportunity NYC- 
Family Rewards was launched, no comprehensive CCT program had been attempted in a high- 
er-income country. 

Built on Mexico’s pioneering Oportunidades program and sponsored by New York City’s Cen- 
ter for Economic Opportunity (CEO), Family Rewards used foundation funding to offer condi- 
tional cash incentives to poor families for a period of three years. Family Rewards was intended 
to help low-income families reduce economic hardship in the short run and to escape intergen- 
erational poverty in the long run, while also incorporating the principle of reciprocity that has 
historically been embedded in the nation’s major income support programs. It did this by offer- 
ing cash incentives to families if they took steps to improve their children’s educational out- 
comes, family members’ preventive health care practices, and parents’ employment. Condition- 
ing transfer benefits is always controversial. But much of America’s safety net (including the 
Earned Income Tax Credit) already conditions transfers on work efforts. In a weak labor mar- 
ket, there may be value in giving low-income families additional opportunities to qualify for 
income transfers while also enabling them to invest in their own futures. 

This interim report on effects through the third year of the program’s operation found that, 
while it operated, Family Rewards continued to reduce poverty and material hardship, increased 
savings by some families, and had some sustained, positive impacts on educational outcomes 
for better-prepared high school students. Particularly encouraging, it increased on-time gradua- 
tion rates for ninth-graders who were academically proficient readers when they entered the 
program. The program also increased families’ receipt of dental care. But its effects on poverty 
and hardship began to fade once the reward payments were no longer available. The program 
also had no effects on younger students’ educational progress or families’ overall health out- 
comes, and it had small effects on parents’ labor market outcomes. 

Recognizing from the early evidence that the program’s original design had produced mixed ef- 
fects, MDRC and CEO revised the Family Rewards model and launched a separate follow-up 
demonstration project in Memphis, Tennessee, and the Bronx, New York. The new model tar- 
gets low-income families with children in grades 9 and 10 only, rather than including children in 
elementary and middle school, as in the original program. It offers fewer rewards, disburses 
payments more frequently, and rewards report card grades in addition to attendance and test 
scores to provide a more immediate incentive for better school performance. It also adds a family 
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guidance component. It is hoped that this refined version of Family Rewards will be more effec- 
tive than the original program. 

As the current evaluation continues and the new Family Rewards program matures, we will 
gain evidence about whether and how CCT programs can be effective in various contexts. In the 
end, we hope to continue to leam whether this approach can reduce long-term and intergenera- 
tional poverty so that future generations can look forward to a better inheritance. 

Gordon L. Berlin 
President, MDRC 
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Executive Summary 


In 2010, the operational phase of New York City’s first experiment testing a comprehensive 
conditional cash transfer (CCT) program, known as Opportunity NYC-Family Rewards, 
concluded — as scheduled — after a planned three-year run. Launched in 2007 as a privately 
funded initiative in six of New York City’s highest-poverty communities, Family Rewards 
aimed to help families break the cycle of intergenerational poverty. This report presents interim 
results on the program’s effects through its final year of operations and, on some measures, 
during the first year after the program concluded. A future report will present longer-term post- 
program results. 

CCT programs transfer cash to poor families to reduce immediate hardship and poverty. 
They condition the cash transfers on families’ efforts to improve their “human capital” (typical- 
ly, children’s educational achievement and family health) in the hope of also reducing intergen- 
erational poverty. Such programs have grown rapidly across lower- and middle-income coun- 
tries, and evaluations have found some important successes. Family Rewards is the first 
comprehensive CCT program to be attempted in a higher-income country. 

Family Rewards tied cash rewards to a prespecified set of activities and outcomes in 
three domains: children’s education, family preventive health care, and parents’ employment. 
The program was available to about 2,400 families for three years. Inspired by Mexico’s 
pioneering Oportunidades program, which offers poor families cash payments that are linked 
primarily to their children’s school attendance and family members’ preventive health care, 
Family Rewards’ effects are being measured via a randomized control trial — meaning that 
eligible families were assigned at random to either a program group, which received the Family 
Rewards intervention, or a control group, which did not. 

The Family Rewards demonstration is one of about 50 initiatives sponsored by New 
York City’s Center for Economic Opportunity (CEO), a unit within the Office of Mayor 
Michael R. Bloomberg that is responsible for testing innovative strategies to reduce the number 
of New Yorkers who are living in poverty. Two national, New York-based nonprofit organiza- 
tions — MDRC, a nonpartisan, social policy research firm, and Seedco, a workforce and 
economic development organization — worked in close partnership with CEO to design the 
demonstration. Seedco, together with a small network of local, community-based organizations, 
operated Family Rewards. In addition to managing the overall demonstration, MDRC is 
conducting the evaluation. A consortium of private funders is supporting the project. 1 


'These funders include Bloomberg Philanthropies, The Rockefeller Foundation, The Starr Foundation, 
the Open Society Institute, the Robin Flood Foundation, the Tiger Foundation, The Annie E. Casey 
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This report examines the implementation of the program and families’ responses to it 
through the end of its three years of operations. As noted above, the report also presents interim 
findings on the program’s effects, or “impacts,” on a wide range of outcome measures. For 
some measures, the results cover three years of follow-up after sample members entered the 
study (that is, when they were randomly assigned). These first three years cover the “program 
phase” of the follow-up period. For other measures, the follow-up period is somewhat longer, 
extending into the beginning of the post-program period. The evaluation findings are based on 
analyses of a variety of administrative records data, a survey of parents that was administered 
about 42 months (or three and a half years) after they entered the study, and qualitative in-depth 
interviews with program staff and families. 

Overall, this report shows that Family Rewards made payments to virtually all families. 
It transferred substantial amounts of cash — over $8,700 per family, on average, over the three- 
year period, with many families receiving considerably more. It succeeded in reducing current 
poverty and material hardship (its main short-term goal), but those effects weakened after the 
cash transfers ended. Family Rewards also produced positive effects on some human capital 
outcomes across all three program domains (children’s education, family health care, and 
parents’ work and training), especially for particular subgroups of participants. For example, it 
produced noteworthy effects on education outcomes for better-prepared high school students. 
At the same time, it left many other important outcomes unchanged. 

Opportunity NYC-Family Rewards was the first comprehensive CCT trial in the Unit- 
ed States. Its promising initial effects on poverty reduction and on a number of human capital 
outcomes offer a reason to continue experimenting with this approach. At the same time, 
features of the model that did not work as well point to a number of ways in which the Family 
Rewards approach could be strengthened. Building on the early evidence, MDRC and CEO 
revised the model considerably, and in 2011, MDRC began testing that new version of Family 
Rewards in Memphis, Tennessee, and the Bronx, New York. 


The Original Program Model 

All CCT programs condition immediate poverty relief on families’ investments in human 
capital, especially in children. However, in adopting this core principle, the designers of Family 
Rewards understood that the model and its delivery structure would have to be adapted to suit a 
vastly different social, economic, and policy context than was present in Mexico and other 
middle- and lower-income countries. In New York City, the program attempted to address two- 
generation poverty in an urban setting, in contrast to the Mexican program’s focus on rural 

Foundation, American International Group, the John D. and Catherine T. MacArthur Foundation, and New 
York Community Trust. 
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poverty, and it was layered on top of an already well-developed network of safety net programs 
and policies. 

Like all CCT programs, Family Rewards was based on the assumption that, for a varie- 
ty of reasons, families may underinvest in their own human capital development. That lack of 
investment — while certainly not the only reason for their financial hardship — can make it 
difficult for parents and their children to escape poverty. The cash payments, in addition to 
being a short-term income supplement to reduce hardship immediately, were intended to 
function as enabling resources and as a stimulus to action. As enabling resources, the extra 
money families earned, once it began to accumulate, could help them to support and promote 
their children’s educational progress, obtain preventive health care, and pursue employment 
opportunities. As a stimulus, the reward payments could encourage families to make extra 
investments of time and energy for those purposes. 

Types of Rewards 

New York City’s program included an extensive set of rewards with the following 
conditions: 

• Education-focused conditions, which included meeting goals for chil- 
dren’s attendance in school, achievement levels on standardized tests, and 
other school progress markers, as well as parents’ engagement with their 
children’s education 

• Health-focused conditions, which included maintaining health insurance 
coverage for parents and their children, as well as obtaining age- 
appropriate preventive medical and dental checkups for each family 
member 

• Workforce-focused conditions, aimed at parents, which included sus- 
taining full-time work and participation in approved education or job 
training activities 

The program offered a set of 22 different incentives during its first two years (some of 
which were discontinued in Year 3), ranging in value from $20 to $600 each per year. (See 
Table ES.l for a detailed list.) The program designers included this broad range to create 
opportunities to assess which incentives might be the most effective. In addition, they sought to 
give families many different ways in which to earn money and to avoid attaching overly large 
amounts of money to any given activity or outcome. After reviewing early evidence of impacts, 
several rewards were discontinued for the third year. This was done to simplify the program, 
lower its costs, and make it easier to replicate should it prove to be successful. 
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The Opportunity NYC Demonstration: Family Rewards 

Table ES.l 


Schedule of Rewards 


Activity 

Reward Amount 

Education incentives 


Elementary and middle school students 


Attends 95% of scheduled school days 
(discontinued after Year 2) 

$25 per month 

Scores at proficiency level (or improves) on annual 
math and English language arts (ELA) tests 
Elementary school students 
Middle school students 

$300 per math test; $300 per ELA test 
$350 per math test; $350 per ELA test 

Parent reviews low-stakes interim tests 
(discontinued after Year 1) 

$25 for parents to download, print, and 
review results (up to 5 times per year) 

Parent discusses annual math and ELA test results 
with teachers (discontinued after Year 2) 

$25 (up to 2 tests per year) 

High school students 


Attends 95% of scheduled school days 
Accumulates 1 1 course credits per year 
Passes Regents exams 
Takes PSAT test 
Graduates from high school 

$50 per month 
$600 

$600 per exam passed (up to 5 exams) 
$50 for taking the test (up to 2 times) 
$400 bonus 

All grades 


Parent attends parent-teacher conferences 

Child obtains library card ( discontinued after Year 2) 

$25 per conference (up to 2 times per year) 
$50 once during program 

Health incentives 


Maintaining public or private health insurance 
(discontinued after Year 2) 

For each parent covered 
If all children are covered 

Per month: $20 (public); $50 (private) 
Per month: $20 (public); $50 (private) 

Annual medical checkup 

$200 per family member (once per year) 

Doctor-recommended follow-up visit 
(discontinued after Year 2) 

$100 per family member (once per year) 

Early-intervention evaluation for child under 30 months 
old, if advised by pediatrician 

$200 per child (once per year) 

Preventive dental care (cleaning/checkup) 

Workforce incentives 

$100 per family member (once per year for 
children 1-5 years old; twice per year for 
family members 6 years of age or older) 

Sustained full-time employment 

$150 per month 

Education and training while employed at least 10 hours per 

week (employment requirement discontinued after Year 2) 

Amount varies by length of course, up to a 
maximum of $3,000 over 3 years 
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The program allowed families to receive cash rewards totaling several thousand dollars 
per year over a three-year period. The actual amounts that families received depended on the 
number and particular type of rewards they earned. (Some rewards carried higher payments than 
others.) Larger families could earn higher payments because each child’s actions could earn 
education and health rewards. 

In general, payments were made directly to the parents. However, some education- 
related payments for high school students were paid directly to the students. Depending on the 
reward, the entire payment was made to the student (for example, for passing a Regents exam) or 
split with the parents (for example, for meeting the attendance standard). To maximize the 
potential incentive value of the rewards, the program imposed no restrictions on how families 
could spend the money. 

The Family Rewards model differs in important ways from CCT approaches in other 
countries. In many countries, CCT programs function as the main government-sponsored safety 
net, or as an important component of it, and they most commonly tie the payments only to 
children’s school enrollment and attendance and to routine health checkups. In contrast, Family 
Rewards included many more conditions and rewards. In the education domain, it was unusual 
in rewarding children’s school achievement, including standardized test score results, not just 
school enrollment and attendance. Its work-related component for parents was also distinctive. 
And as a short-term intervention layered on top of an already well-developed social safety net, 
Family Rewards served as a supplemental program rather than as the core welfare system, in 
contrast to programs in Mexico and a number of other countries. It was also unusual in being 
operated by private, nonprofit agencies rather than by the government. 

The Delivery Structure 

Seedco, the main implementing agency, assembled a network of local organizations in 
the designated community districts to assist in implementing Family Rewards. Called “Neigh- 
borhood Partner Organizations” (NPOs), these agencies recruited and enrolled eligible families 
into the research sample and served as the face of the program in the communities . 2 They 
provided ongoing customer service to participants who requested assistance, such as in making 
claims for the rewards or for infonnation about other services in the community. NPOs also 
conducted informational workshops on how to earn and claim rewards in each of the domains in 
which the incentives were offered. Seedco maintained a telephone helpline and a Web site to 
provide additional infonnation and assistance to families. 


2 These organizations are Urban Health Plan and BronxWorks (formerly Citizens Advice Bureau) in the 
Bronx; Brownsville Multi-Service Center and Groundwork, Inc., in Brooklyn; and Catholic Charities and 
Union Settlement Association in Manhattan. 
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Once Seedco verified that families earned rewards (which it did using a combination of 
administrative data from city agencies and special “coupon book” forms submitted directly by 
participants), it initiated a process of transferring payments electronically into participants’ 
newly opened or existing bank accounts or, if they preferred, onto stored value cards (which are 
prepaid cards, like gift cards). To provide families with a safe banking option, New York City 
officials worked with several banks and credit unions to develop special “Opportunity NYC 
accounts” that carried no fees and came with debit cards that were impossible to overdraw. The 
reward payments were made every two months, and families could access the money at any 
time through any automatic teller machine (ATM). 

Envisioned as an “incentives-only” intervention, the program model did not provide so- 
cial services or case management. However, it did include an infonnation-and-referral compo- 
nent wherein the implementing agencies (Seedco and the NPOs) referred families (upon 
request) to other agencies in the community that provided relevant services. 


The Study Sample 

Family Rewards is being evaluated through a randomized control trial involving approximately 
4,800 families, with 1 1,000 children, who applied to the program. The program could not serve 
all applicants, and the selection of participants was determined on a random basis. Through a 
lottery-like process, half of the applicant families were picked for Family Rewards and offered 
the incentives, and half were assigned to a control group that was not offered the incentives. 
Using such a process helps ensure that the program effects estimated by the evaluation are truly 
a result of the intervention. 

Family Rewards targeted families who lived in selected community districts and who 
had incomes at or below 130 percent of the federal poverty level. Eligible families had to 
have at least one child in the fourth, seventh, or ninth grade. Those grades were selected 
because they are at or near the start of critical transition points in education. Once a family 
volunteered for the study, all children in the family who were school-age or younger were 
eligible for the program. The parents as well as the children had to be legal residents of the 
United States in order to be eligible. 

Program operations began with the start of the new school year in September 2007. To 
ensure that the program reached a broad cross-section of children, not just the most motivated 
and active, potentially eligible families living in the targeted communities were identified from 
lists of students in the free school lunch program maintained by the New York City Department 
of Education. Seedco and the NPOs then attempted to recruit a representative group of those 
families through mailings, phone calls, and home visits, inviting them to apply to be in the 
study. Those who agreed were randomly assigned to the program or control group. 
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Implementation and Reward Receipt 

A prior MDRC report examined the first two years of program operation in depth. 3 It showed 
that by the second year, although complex to administer, the program was being operated in a 
way that was generally consistent with its designers’ vision. 

Program operations remained strong in the third and final year, which ended in August 
2010. During that year, staff also began to focus on an “exit strategy” to prepare families to 
cope with the ending of the reward payments. Given the relatively short period of the program 
and the fact that families would be exiting in the wake of the Great Recession, most participants 
were likely to see their income drop as the program came to an end. The staff tried to help 
participants prepare for this income cliff by encouraging them to increase their labor market 
earnings and adjust their consumption patterns. 

In-depth interviews with a sample of participants suggest that families reacted to the 
end of the program with acceptance. They expressed gratitude for having had the experience, 
but some were doubtful that they could replace the lost income with earnings from employment. 
Some expected to draw more on savings that they had accumulated during the program, and 
some expressed an intention to go back to school or try to increase their wage earnings. 

• Overall, families earned a substantial amount of reward money from the 
program — an average of over $8,700 for all three years combined. 

Virtually all families earned at least some rewards during the three program years, and 
89 percent earned at least one reward in Year 3 (when fewer rewards were offered). Reward 
amounts averaged over $3,100 during each of the first two years and $2,700 in the third year 
(when several rewards were discontinued). A majority of families — approximately 57 percent 
— earned at least $7,000 over the life of the program. The top 20 percent earned more than 
$13,000 in reward money. 

To put these amounts in perspective, the federal poverty level for a family of three (for 
example, a single parent with two children) in 2009 (roughly midway through the program 
period) was $18,310. Thus, families of that size and income level who received $3,000 in 
reward payments in a year would increase their annual income by about 16 percent. Similarly 
sized families with income that is below half of the poverty level (or below $9,155 for the 
example cited above), which some experts would define as living in “severe poverty,” would 
boost their income by 33 percent. Or, put differently, a reward amount of $3,000 would add 


3 James Riccio, Nadine Dechausay, David Greenberg, Cynthia Miller, Zawadi Rucks, and Nandita Ver- 
ma, Toward Reduced Poverty Across Generations: Early Findings from New York City ’s Conditional Cash 
Transfer Program (New York: MDRC, 2010). 
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about 21 percent to the total wages ($14,560) of a single parent who was paid $8 per hour for 
working 35 hours per week for an entire year. 

Compared with other families, those in the top 20 percent of earners were larger (giving 
them more opportunities to earn rewards) and tended to be less disadvantaged. For example, the 
parents were more educated, more likely to be employed, and more likely to be married, and the 
families were less likely to be receiving government transfer benefits. In addition, in-depth 
interviews suggest that parents who were top earners may have been better organized, more able 
to handle the verification procedures associated with the program, and more likely to track their 
families’ performance against the conditions they needed to meet in order to earn rewards. 

Most reward money came from the education domain, accounting for 45 percent of the 
$20.6 million spent on reward payments over the full course of the program. Health care 
rewards accounted for 34 percent of total payments, and workforce rewards (primarily for full- 
time work rather than education or training) accounted for 21 percent. Virtually 100 percent of 
families earned at least one education and one health reward, while about 53 percent earned a 
workforce reward. 

• Parents used the reward money to pay for basic household expenses, 
some “extras,” and, in some cases, to save for college and pay for special 
lessons to help their children in school. 

Family Rewards imposed no restrictions on families’ access to their reward money or 
how they could spend it, and throughout the program families used the extra money in a variety 
of ways. Common uses included paying for basic living expenses, paying off bills, paying for 
school-related supplies or activities, buying electronic goods, saving for the future, and covering 
special recreational outings for the family, sometimes as a reward for school accomplishments. 
For many families, celebration of accomplishments took the form of spending time together on 
leisure activities, like eating out, going on a trip, or seeing a movie that would otherwise have 
been prohibitively expensive. 

High school students received substantial amounts of money in their own bank ac- 
counts for meeting education-related conditions. A companion study of high school students 
and their parents found that parents exercised varying degrees of control over how much 
access students had to their rewards. 4 The vast majority of parents who were interviewed for 
the 42-month survey (72 percent) said that their high school-age child had to ask them for 
permission to spend the money. Only 17 percent gave their children freedom to spend the 


4 Pamela Morris, J. Lawrence Aber, Sharon Wolf, and Juliette Berg, Using Incentives to Change How 
Teenagers Spend Their Time: The Effects of New York City’s Conditional Cash Transfer Program (New 
York: MDRC, 2012). 
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money as they wished, and 9 percent did not allow their high school-age children to spend it 
at all. Despite the sizable money transfers into students’ own accounts, the program did not 
increase parent-teenager conflict, a problem that some observers feared. In addition, the 
program may have reduced certain troublesome behaviors among the teenagers, such as 
aggression and substance abuse. 


Interim Impacts 

Findings on the program’s effects, or “impacts,” are available on a wide variety of outcome 
measures covering three to four years after each family’s time of entry into the study, depending 
on the data source. Thus, the results reported here provide a full picture of the program’s effects 
while families were still participating in it and soon after the program ended. (Longer-tenn post- 
program impact findings will be presented in a future report.) All impacts that are discussed in 
this summary are statistically significant unless otherwise noted, thus indicating a high degree of 
confidence that the observed differences between program and control groups are most likely a 
result of the program rather than of chance. 5 

• Family Rewards reduced families’ current poverty and economic hard- 
ships, including difficulties securing enough food and some housing- 
related hardships. 

As it is for all CCT programs, reducing current poverty and hardship was a key short- 
tenn objective of Family Rewards. In this area, Family Rewards succeeded. It substantially 
improved families’ economic position while they were in the program. For example, counting 
the value of the reward payments, it boosted self-reported average monthly household income 
for the program group by $353 in Year 3, an improvement of about 22 percent relative to the 
control group’s average monthly income of $1,620. (See Table ES.2.) This extra income reduced 
the proportion of families living at or below the federal poverty level by 12 percentage points 
below the control group rate of 68 percent. 6 The program also cut the proportion of families 


5 Nonetheless, impact estimates are calculated for a large number of outcome variables, raising the risk 
of finding statistically significant effects just by chance. No formal statistical controls were used to guard 
against this risk, and caution should be used in attributing meaning to isolated impacts that are not part of a 
broader pattern of effects. 

6 ln this study, income and poverty estimates include self-reported monthly cash income plus the cash 
value of benefits from the Food Stamp Program, now called the Supplemental Nutrition Assistance Program 
(SNAP), but it excludes tax credits. Poverty estimates are based on comparisons with the official federal 
poverty levels for families of various sizes. The reward payments did not affect other public benefits that 
families may have been receiving, such as SNAP, welfare payments under the Temporary Assistance for 
Needy Families (TANF) program, Medicaid, housing subsidies, and the Earned Income Tax Credit. 
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The Opportunity NYC Demonstration: Family Rewards 
Table ES.2 


Impacts on Selected Outcomes Measuring Poverty, Material Hardship, 
Banking, Health Care, and Employment Through the Final Program 
Year or Early Post-Program Period 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Change 

(%) 

Income and Dovertv 

Household income during Year 3 
(including Family Rewards payments) 
Average monthly ineome a,b,c ($) 

1,973 

1,620 

353 *** 

21.8 

Annual income at or below federal poverty level a,b (%) 

56.0 

68.2 

-12.2 *** 

-17.8 

Annual income less than 50% of federal 
poverty level a,b (%) 

16.3 

27.4 

J H=H=H= 

-40.7 

Household income during early post-program period 
(excluding Family Rewards payments) 

Average monthly income a,d ($) 

1,700 

1,620 

79 * 

4.9 

Annual income at or below federal poverty level a,d (%) 

66.2 

68.2 

-2.0 

- 

Annual income less than 50% of federal 
poverty level J,d (%) 

25.9 

27.4 

-1.5 

— 

Material hardship (%) 

Family "sometimes" or "often" did not have enough 
food to eat in past month 

15.3 

20.7 

_5 4 *** 

-26.2 

Family usually did not have enough money to make 
ends meet at end of month 

35.4 

41.0 

-5.6 *** 

-13.7 

Family did not pay full rent or mortgage in past year c 

40.0 

44.1 

-4.2 * 

-9.4 

Parent agrees "strongly" or "somewhat" that current 
financial situation is "better than last year" 

51.4 

46.6 

4.8 ** 

10.3 

Banking and savings (%) 

Parent currently has any bank account 

64.0 

46.6 

17 ^ *** 

37.5 

Parent cashes check at check casher at least 
once a month 

29.2 

31.5 

-2.3 



Family has any savings 

24.6 

16.8 

7 g *** 

46.8 

Family's average savings exceed $500 

12.5 

9.2 

3 2** 

35.1 

Parent borrows cash from family or friends 

47.3 

52.5 

-5.2 ** 

-9.8 


(continued) 
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Table ES.2 (continued) 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Change 

(%) 

Parents' use of health services and health status 

Had a period with no health insurance 
coverage in past 12 months 

15.3 

17.6 

-2.3 * 

-13.3 

Uses hospital emergency room as usual source 
of care when sick 

3.0 

4.0 

- 1.0 

— 

Has seen health professional for any reason 
in past 12 months 

94.4 

94.5 

- 0.1 



Had a health checkup in past 12 months 

90.0 

88.9 

1.1 

- 

Treated for any medical condition 

50.1 

47.8 

2.3 

- 

Self-rated health is "excellent" or "very good" 

53.2 

48.7 

4.5 

- 

Has seen dentist for any reason in past 12 months 

85.4 

75.3 

J *** 

13.4 

Had 2 or more dental checkups in past 12 months 

45.2 

33.5 

J J g *** 

35.2 

High school students' use of health services (%) 

Uses hospital emergency room as usual source 
of care when sick 

2.5 

3.2 

- 0.8 


Has seen dentist for any reason in past 12 months 

93.8 

89.1 

47 ** 

5.2 

Had 2 or more dental checkups in past 12 months 

62.9 

44.1 

18.8 *** 

42.6 

Parents' emplovment outcomes 

Employment status, survey (%) 

Currently employed at the time of the survey 

56.0 

49.6 

0 4 *** 

12.9 

f 

Working full time (at least 30 hours per week) 

44.4 

39.5 

4 9 *** 

12.4 

Employment status, Ul records 
Ever employed, Y ear 3 (%) 

52.5 

53.3 

-0.9 


Average quarterly employment, Y ear 3 (%) 

46.1 

46.7 

- 0.6 

- 

Average earnings, Year 3 ($) 

12,414 

12,529 

-116 

— 


SOURCE: MDRC calculations using data from the Family Rewards 42-month survey and New York 
State unemployment insurance (UI) wage records. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the difference between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 
5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. 

(continued) 
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Table ES.2 (continued) 


Rounding may cause slight discrepancies in calculating sums and differences. 

Percentage change shown only for statistically significant impacts. 

Unless otherwise specified, the survey measures refer to the early post-program period. 

Dollar averages include zero values for sample members who were not employed. 

U1 records include only employment and earnings in jobs covered by the New York State U1 
program. They do not include employment outside of New York State, nor in jobs not covered by the 
UI system (for example, "off-the-books" jobs and federal government jobs). 

““Monthly household income amounts equal to or greater than $10,000 were excluded from this 
calculation. About 7.2 percent of the sample is excluded from the income measures because 
respondents did not know or refused to provide the information. An additional 0.6 percent of the 
sample was excluded because the income provided was over $10,000. 

b Annual household income is calculated by multiplying by 12 the respondent's income in the 
month prior to the survey interview. For program group members, it includes Family Rewards 
payments earned during program Year 3. The federal poverty level was calculated based on annual 
income (monthly income multiplied by 12) and the household size at the time of the survey. The 
poverty threshold was measured according to the 2010 or 201 1 Poverty Guidelines, depending on 
when a respondent was interviewed. 

c The Year 1 income measures reported on the 18-month survey are within 3 percent of the 42- 
month income measures reported here. 

d Annual household income is calculated by multiplying by 12 the respondent's income in the 
month prior to the survey interview. This calculation does not include Family Rewards payments 
earned during program Year 3. The federal poverty level was calculated based on annual income 
(monthly income multiplied by 12) and the household size at the time of the survey. The poverty 
threshold was measured according to the 2010 or 201 1 Poverty Guidelines, depending on when a 
respondent was interviewed. 

c Only about 4 percent of the survey sample (N = 130) owned an apartment or a house at the time 
of the survey. 

f If a respondent worked multiple jobs at the time of the interview, then only the characteristics of 
the primary job are reported. (The job at which the respondent worked the most hours is considered 
primary.) 


who were living in severe poverty (that is, families with starting incomes less than 50 percent of 
the federal poverty level, who make up about half of the study sample). 

Most of these poverty reductions are attributable to the cash transfers that families re- 
ceived, rather than to increased earnings from jobs. Once the program ended and the transfers 
were no longer available, families’ incomes dropped and were not substantially different from 
those of the control group. 

The extra income they received during the program period helped families reduce a va- 
riety of material hardships, and those effects persisted into the early post-program period. For 
example, the proportion of families who experienced “food insufficiency” (as indicated by 
parents responding on the 42-month survey that their families “sometimes” or “often times” did 
not have enough to eat) dropped from over 20 percent in the control group to about 15 percent 


ES-12 



in the program group, a reduction of over 5 percentage points. 7 Program group families were 
less likely than the control group to report not having enough money to pay their rent some time 
in the past year. They were more likely to report having enough money to “make ends meet” 
and that their financial situation had improved over the prior year. 

The reductions in hardships were largely concentrated among families who were living 
in severe poverty at the time they entered the program. Among that group, the program caused a 
9 percentage point reduction in the likelihood of reporting food insufficiency after the program 
ended, and about an 1 1 percentage point reduction in the likelihood of not paying their full rent 
in the past year (not shown in table). 

• Family Rewards helped parents increase their savings and reduce their 
reliance on families and friends for cash loans. 

The parents in Family Rewards were about 18 percentage points more likely than the 
control group (64 percent versus 47 percent) to report having a bank account after the program 
had ended. They were 8 percentage points more likely than the control group (25 percent versus 
17 percent) to have any savings. They were also more likely to have savings of more than $500, 
and less likely to borrow cash from family or friends. 

Education 

• Family Rewards did not improve school outcomes for elementary or 
middle school students. 

The analysis examined the effects of Family Rewards on school attendance rates, grade 
progression, and various achievement measures during the three years of the program and one 
year afterward — or four years in total after students began the program. 

For elementary and middle school students, the analysis found few positive effects on 
attendance rates, scores on standardized tests, or other school outcomes during the program 
period or by the end of Year 4. In addition, subgroup analyses did not reveal any consistent 
patterns of positive effects for particular types of students in those grades. Perhaps the model’s 
limited approach for these children — of rewarding only attendance (which was already high, 
leaving little room for improvement) and standardized test scores (rather than more immediate 
performance indicators, such as good report card grades) — might explain in part why Family 
Rewards did not have an educational payoff for this group. 


7 Slight discrepancies in percentages are a result of rounding. 
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• Family Rewards had few effects on school outcomes for high school stu- 
dents overall. However, it substantially increased graduation rates and 
other outcomes for students who were already stronger readers. 

Students who were behind educationally when they entered Family Rewards did not 
experience educational gains from the program. In contrast, those who entered better prepared 
for high school — who may have been in a better position to take advantage of the incentives 
offer — do appear to have benefited. Although subgroup findings tend to carry less statistical 
certainty than full-sample results, a number of other studies of education-focused incentives 
programs have similarly found more positive effects for more capable students. 8 

Family Rewards had particularly strong effects on students in the ninth-grade cohort 
who had scored at or above the basic proficiency level on their eighth-grade standardized 
English language arts (ELA) test (which primarily tests reading skills) before random assign- 
ment. For this subgroup, which made up almost one-third of the overall sample of ninth-graders, 
Family Rewards appears to have improved a range of school outcomes. (See Table ES.3.) These 
include an 8 percentage point increase in the likelihood of graduating from high school within 
four years (a gain of 12 percent above the 67 percent graduation rate among control group 
students who were ELA-proficient at the beginning of the study). The program also produced a 
10 percentage point increase in the proportion of ELA-proficient students who were enrolled in 
grade 12 in Year 4, indicating that they were progressing through high school at the expected 
rate. In addition, Family Rewards increased their likelihood of earning at least 44 credits (the 
amount needed to graduate) by 9.6 percentage points, and the likelihood of passing at least five 
New York State Regents exams by 9.5 percentage points. 9 These effects are particularly 
noteworthy because they occurred without any changes in the schools themselves or in teachers’ 
instructional practices. 

For the ninth-graders who were proficient on their eighth-grade math test, Family Re- 
wards produced positive effects on various educational outcomes during the program phase 
only. For example, as Table ES.3 shows, it improved their attendance rates and credit accumu- 
lation while they were in the program. However, these positive effects did not persist into Year 
4, when the incentives were no longer available. In addition, the math-proficient subgroup did 
not experience an increase in on-time graduation. 


8 See, for example, Joshua Angrist and Victor Lavy, “The Effects of High Stakes High School 
Achievement Awards: Evidence from a Randomized Trial,” American Economic Review 99, 4 (2009): 
1384-1414. 

"’Students must pass at least five tests in specified subject areas in order to graduate with a diploma rec- 
ognized by the New York State Board of Regents, which sets standards and regulations for all public 
schools in the state. 
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The Opportunity NYC Demonstration: Family Rewards 
Table ES.3 


Impacts on Selected Education Outcomes for Students in Grade 9 


Grade Level and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Change 

(%) 

Students in grade 9 at baseline 





Graduated within 4 years (%) 

49.2 

48.2 

1.1 

— 

Enrolled in any grade in Y ear 4 (%) 

80.1 

79.2 

0.9 

- 

Enrolled in grade 12, Year 4 (%) 

53.1 

51.2 

1.9 

- 

Average attendance rate, Year 3 (%) 

69.4 

67.7 

1.7 

- 

Average attendance rate, Year 4 (%) 

60.7 

59.7 

1.1 

- 

Attendance rate 95% or higher, Year 3 (%) 

25.1 

21.9 

3.1 * 

14.3 

Attendance rate 95% or higher, Y ear 4 (%) 

17.4 

15.3 

2.1 

- 

Average number of credits earned, Years 1 to 4 

32.7 

31.9 

0.8 

- 

Earned at least 33 credits, Years 1 to 3 (%) 

41.9 

40.9 

0.9 

- 

Earned at least 44 credits, Years 1 to 4 (%) 

41.5 

40.5 

0.9 

— 

Passed at least 5 Regents exams, Y ears 1 to 4 (%) 

36.7 

35.7 

1.1 

- 

Students in grade 9 at baseline, bv proficiency level 





on 8 th grade English language arts (ELA) test 8 





Graduated within 4 years (%) 



tt 


Proficient on 8th grade ELA test 

74.8 

66.9 

8.0 ** 

11.9 

Not proficient on 8th grade ELA test 

43.2 

45.9 

-2.8 

- 

Enrolled in any grade in Y ear 4 (%) 





Proficient on 8th grade ELA test 

90.9 

89.0 

1.9 

- 

Not proficient on 8th grade ELA test 

82.7 

81.3 

1.4 

- 

Enrolled in grade 12, Year 4 (%) 



tt 


Proficient on 8th grade ELA test 

78.4 

68.2 

10 i *** 

14.8 

Not proficient on 8th grade ELA test 

48.3 

50.3 

-2.0 

- 

Average attendance rate, Y ear 3 (%) 



tt 


Proficient on 8th grade ELA test 

83.9 

77.1 

6.8 *** 

8.8 

Not proficient on 8th grade ELA test 

68.1 

67.7 

0.4 

- 

Average attendance rate, Year 4 (%) 





Proficient on 8th grade ELA test 

76.7 

71.6 

5.1 * 

7.1 

Not proficient on 8th grade ELA test 

59.8 

60.4 

-0.6 

- 

Average number of credits earned, Years 1 to 4 



ttt 


Proficient on 8th grade ELA test 

44.3 

40.0 

4 3 *** 

10.8 

Not proficient on 8th grade ELA test 

30.9 

31.8 

-0.8 

- 

Earned at least 33 credits, Years 1 to 3 (%) 



ttt 


Proficient on 8th grade ELA test 

68.4 

55.4 

13.0 *** 

23.4 

Not proficient on 8th grade ELA test 

36.3 

40.1 

-3.8 

- 
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Table ES.3 (continued) 


Outcome (%) 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Change 

(%) 

Earned at least 44 credits, Years 1 to 4 (%) 
Proficient on 8th grade ELA test 

66.1 

56.6 

tt 

9.6 ** 

16.9 

Not proficient on 8th grade ELA test 

36.9 

39.8 

-2.8 

- 

Passed at least 5 Regents exams, Y ears 1 to 4 (%) 
Proficient on 8th grade ELA test 

72.5 

63.1 

ttt 

9.5 ** 

15.1 

Not proficient on 8th grade ELA test 

25.8 

28.9 

-3.1 

- 

Students in grade 9 at baseline, bv proficiency 
level on 8th grade math test a 

Graduated within 4 years (%) 

Proficient on 8th grade math test 

74.8 

71.3 

3.5 

- 

Not proficient on 8th grade math test 

41.7 

42.2 

-0.5 

- 

Enrolled in any grade in Y ear 4 (%) 

Proficient on 8th grade math test 

91.1 

89.3 

1.8 

- 

Not proficient on 8th grade math test 

81.9 

80.5 

1.4 

- 

Enrolled in grade 12, Year 4 (%) 

Proficient on 8th grade math test 

77.6 

71.8 

5.8 

- 

Not proficient on 8th grade math test 

47.5 

47.0 

0.5 

- 

Average attendance rate, Y ear 3 (%) 

Proficient on 8th grade math test 

82.8 

78.2 

4.6 ** 

5.9 

Not proficient on 8th grade math test 

67.8 

66.7 

1.1 

- 

Average attendance rate, Year 4 (%) 

Proficient on 8th grade math test 

75.6 

72.6 

3.0 

- 

Not proficient on 8th grade math test 

59.2 

58.9 

0.3 

- 

Average number of credits earned, Years 1 to 4 

Proficient on 8th grade math test 

43.5 

41.3 

2.3 * 

5.5 

Not proficient on 8th grade math test 

30.8 

30.6 

0.2 

- 

Earned at least 33 credits, Years 1 to 3 (%) 
Proficient on 8th grade math test 

67.4 

59.8 

t 

7.6 * 

12.7 

Not proficient on 8th grade math test 

35.6 

37.0 

-1.4 

- 

Earned at least 44 credits, Years 1 to 4 (%) 

Proficient on 8th grade math test 

64.4 

60.4 

4.1 

- 

Not proficient on 8th grade math test 

36.9 

36.8 

0.1 

- 

Passed at least 5 Regents exams, Y ears 1 to 4 (%) 

Proficient on 8th grade math test 

69.5 

68.8 

0.7 

- 

Not proficient on 8th grade math test 

25.6 

24.0 

1.6 

- 
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Table ES.3 (continued) 


SOURCE: MDRC calculations using data from New York City Department of Education 
administrative records. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and 
control groups. The p-value indicates the likelihood that the difference between the program and 
control groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 
percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical 
significance levels are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. Standard errors were adjusted to account for multiple 
observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Percentage change is shown only for statistically significant impacts. 

Years 1, 2, 3, and 4 cover the 2007-2008, 2008-2009, 2009-2010, and 2010-2011 school years, 
respectively. 

The Regents measures in this table include the following Regents exams: English, Math A, 
Math B, Geometry, Integrated Algebra, Algebra 2/Trigonometry, U.S. History and Government, 
Global History and Geography, Living Environment, Chemistry, Physics, and Earth Science. 

a ln New York State, students who score at a level of 3 or higher on a 4-point scale are deemed 
"proficient." 


When they entered the study, ninth-graders who were proficient in reading were not 
necessarily proficient in math, and vice-versa. 10 Why Family Rewards would have stronger and 
more sustained effects for reading-proficient students is not immediately clear. This question 
remains an important topic for further investigation. 

Family Rewards did not help students who were less prepared for high school. More 
specifically, the analysis found no pattern of statistically significant impacts on educational 
outcomes for students in the ninth-grade cohort who had scored below the proficiency threshold 
on either the eighth- grade ELA or math exams before random assignment. 11 

Some critics of incentives worry that extrinsic rewards can reduce children’s intrinsic 
motivation to leam, especially after the incentives end, and thus harm their educational out- 
comes. So far, there is no indication that Family Rewards has caused any consistent pattern of 
statistically significant negative effects on school outcomes, even after the program ended. The 
evaluation will continue to assess that risk once longer-term data are available. 

l0 On the students’ eighth-grade standardized tests, the correlation between ELA proficiency and math 
proficiency was about 0.44. 

n As Table ES.3 shows, on most outcome measures, the impacts for ELA-proficient students were sta- 
tistically significantly different from the impacts for non-ELA-proficient students. 
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Health 

• Family Rewards did not increase fam il ies’ use of preventive medical 
care, which was already high, and it had few effects on health outcomes. 

The health-related incentives of the Family Rewards program were designed to encour- 
age low-income families to adopt better preventive health care practices. It turned out that a 
higher proportion of families than the program’s designers had expected were already receiving 
health insurance coverage and practicing preventive health care. This finding may reflect the 
success of efforts by New York State and New York City to expand access to health coverage 
in the years leading up to and during the study period. 12 

Perhaps for that reason, Family Rewards had few noteworthy health-related impacts, 
according to the 42 -month survey. (See Table ES.2.) It did reduce the likelihood that parents or 
their children would experience an interruption in health insurance coverage in the prior year by 
over 2 percentage points (even after the health insurance rewards were discontinued in the third 
program year). But it did not improve the likelihood that parents or children would get health 
checkups or see health professionals for other reasons, primarily because most families already 
received those services. A small reduction in families’ use of emergency rooms for routine 
medical care that was evident from the evaluation’s 18 -month survey faded, with very few 
families in either the program or control groups reporting on the 42 -month survey that they had 
used emergency rooms for that purpose. 

Although Family Rewards did not lead to improvements on a range of parents’ health 
outcomes, or on health outcomes that parents reported for their children, one noteworthy 
suggestive subgroup finding emerged. Parents who indicated at the time of random assignment 
that they were in “fair” or “poor” health (about 20 percent of the sample; not shown in Table 
ES.2) were 6.2 percentage points more likely than the control group (or almost twice as likely) 
to report that they were in “very good” or “excellent” health at the time of the 42-month survey. 
They also reported lower rates of asthma. 13 Although the latter finding is not easily explained by 
other patterns in the data, it may be a topic worthy of further exploration in future studies. 

• Family Rewards produced large increases in fam il ies’ use of dental care 
services. 


12 The study sample did not include low-income single adults or undocumented immigrants, who are 
much less likely to have health insurance. 

l3 The difference in impacts on asthma across the health subgroup categories is statistically significant. 
The difference in impacts on current self-reported health is in the same direction but is not statistically 
significant, making the finding less certain. 
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Family Rewards led to increased dental care for parents and children alike. (See Table 
ES.2.) For example, parents in the program group were 10 percentage points more likely than 
control group parents to report having seen a dentist for any reason in the prior year, and about 12 
percentage points more likely to have had two or more dental checkups in the past year. Strong 
positive effects were also observed among high school students (for example, a 19 percentage 
point increase in two dental checkups in the past year) and among younger children (not shown). 

Employment 

• Family Rewards increased the likelihood of self-reported full-time em- 
ployment. It did not increase employment in or earnings from jobs cov- 
ered by the unemployment insurance (UI) system. 

According to the 42 -month survey of parents, the program increased the likelihood of 
working at the time of the interview by 6 percentage points above the control group rate of 50 
percent. This difference was driven by an increase in full-time work (which the program 
rewarded). (See Table ES.2.) However, the program had no statistically significant impact on 
the average quarterly employment rate in Ul-covered jobs over a three-year follow-up period, 
according to administrative records data. 14 The very small negative effect in Year 3 on average 
earnings in Ul-covered jobs was not statistically significant. 

Some jobs are not covered by the UI system, such as self-employment, federal govern- 
ment employment, and domestic work. In addition, the UI system also misses informal (casual 
or irregular) jobs that are never reported to state agencies. It is not clear why the effects of the 
program would vary across types of employment. Perhaps for some parents, non-UI jobs were 
easier to get in a weak economy, particularly those that offered the full-time hours necessary to 
qualify for the program’s workforce rewards. Such jobs may also have been more attractive 
options if they were more conveniently located, easier to obtain, or offered more flexible 
schedules than Ul-covered jobs. 

It is also not clear why the program did not lead to larger increases in all types of em- 
ployment (including Ul-covered jobs), a finding that stands in contrast to previous work 
incentives programs. It may be that the added income that families received from the education 
and health rewards offset the program’s work incentives for some participants, especially those 
who would have the most difficult time finding jobs in a tough economy. Indeed, subgroup 
analyses found that the program had a statistically significant negative effect on labor market 
outcomes for parents who entered the program with lower education levels and other disad- 
vantages; in other words, they worked and earned less than they would have in the absence 


14 Employers report the wages of workers to the UI system on a quarterly basis. 
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of the program, according to UI records. For example, those without a high school diploma or 
General Educational Development (GED) certificate had an average quarterly employment rate 
in Year 3 that was 3 percentage points lower than that of their counterparts in the control group, 
and they earned an average of $1,790 less (a reduction of almost 8 percent). 15 


Conclusion 

The evidence that is available so far on Family Rewards shows that a CCT approach in one 
large city in a higher-income country can reduce immediate poverty and material hardship and 
promote at least some improvements in some forms of human capital investment, especially for 
certain subgroups. At the same time, the specific model tested in New York City left many 
important outcomes unchanged. 

The evaluation of Family Rewards is continuing and the final story remains to be writ- 
ten. Further evidence will be available in the next evaluation report, to be completed in 2014, 
which will present findings on the program’s effects over five to six years after random assign- 
ment. In the meantime, it seems reasonable to draw at least two general conclusions: (1) the 
Family Rewards model has not demonstrated its value enough to scale it up as a broader 
antipoverty policy in its original form, but (2) because of its success in reducing short-term 
poverty and material hardship while achieving at least some improvements in human capital 
development, continuing to experiment with a CCT approach in the United States has merit. 

With these conclusions in mind, CEO and MDRC joined forces again to design and test 
a “next generation” version of Family Rewards. The new model, referred to as “Family Re- 
wards 2.0,” builds on the lessons of the original New York City demonstration and incorporates 
several important modifications. It was launched in the Bronx, New York, and Memphis, 
Tennessee, in the summer of 201 1 for low-income families with high school students in grades 
9 or 10, all of whom were TANF or SNAP recipients. It includes a streamlined set of financial 
rewards, more frequent payments, and a new family guidance component to try to help more 
parents and students meet the conditions that enable them to earn rewards. 16 It is hoped that 
these refinements to the model will make it a more effective intervention. The project is an 
initiative of the federal Social Innovation Fund, sponsored by the Corporation for National and 
Community Service. Like the original model, Family Rewards 2.0 is being carefully tested 
using a randomized control trial. An initial report on the sites’ operational experiences and 
infonnation on some early impact findings are planned for release in late 2014. 

15 The difference in impacts on average quarterly employment rates for parents with a high school di- 
ploma or GED certificate compared with parents who did not have one of those credentials is statistically 
significant. The difference in earnings impacts across those two education subgroups is in the same direction 
but is not statistically significant. 

16 See Chapter 7 of this report for more information about the new Family Rewards model. 
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Chapter 1 

Introduction 


In 2007, New York City launched, on a pilot basis, an experimental antipoverty initiative called 
Opportunity NYC-Family Rewards, a conditional cash transfer (CCT) program to help families 
break the cycle of intergenerational poverty. As planned, the program offered participants a 
three-year intervention, which concluded, on schedule, in late 2010. A comprehensive evalua- 
tion, using a randomized control trial, is assessing the program’s effects on family poverty and 
well-being over at least five years after families entered the study, including two years after 
families exited the program. This report presents interim findings on the program’s effects on a 
variety of education and family outcomes. The results presented here extend through the 
beginning of the post-program phase. (See Box 1.1 for the demonstration timeline.) 

The first of its kind in the United States, Family Rewards ties cash rewards for very 
low-income families to a variety of activities and outcomes related to children’s educational 
efforts and achievement, family preventive health care practices, and parents’ employment. The 
intent of the program is to reduce family poverty and hardships in the short term while simulta- 
neously supporting and encouraging families to invest in their own health, education, and 
employment potential — or “human capital development” — for their longer-term economic 
security. Family Rewards was inspired by similar initiatives in other countries, particularly 
Mexico’s Oportunidades program, and it is being tested in six of New York’s highest-poverty 
communities. 

Family Rewards is one of about 50 initiatives sponsored by New York City’s Center 
for Economic Opportunity (CEO). A unit within the Office of Mayor Michael R. Bloomberg, 
CEO was established on the recommendation of a special “Poverty Commission” appointed 
by the mayor in 2006 to test a wide range of innovative antipoverty strategies. 1 Family 
Rewards is one of three incentives-based poverty reduction strategies launched by CEO in 
2007. The other two are: 

• Work Rewards, a project that offers work and training incentives to low- 
income recipients of government rent subsidies from New York City’s Hous- 
ing Choice Voucher (Section 8) program 2 


'For more information on CEO and its history and work, see the New York City Center for Economic 
Opportunity Web site at www.nyc.gov/ceo. 

2 Verma et al. (2012). 
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Box 1.1 


Family Rewards Demonstration Timeline 

• Sample recruited: July 2007 to December 2007 

• First program year: September 2007 to August 2008 

• Second program year: September 2008 to August 2009 

• Third program year: September 2009 to August 20 1 0 

• Evaluation reports: 2010 to 2014 

• Social Innovation Fund replication test in New York City and Memphis, Tennessee, 
launched in 20 1 1 


• Spark, a school-based education incentives program that was designed to im- 
prove the school performance of fourth- and seventh-graders by rewarding 
good perfomiance on a series of standardized tests administered over the 
course of the academic year 3 

These three projects differ in important ways, but all offer cash rewards to help low- 
income families build human capital. Together they make up a set of demonstration projects 
known collectively as Opportunity NYC, and each has been part of a rigorous randomized 
control trial evaluation in which study participants are randomly assigned to either a program 
group, which receives the intervention, or a control group, which does not. Random assignment 
helps ensure that observed differences in outcomes between the two groups are truly a result of 
the program. A consortium of private funders has supported these studies. 4 

Two national, New York-based nonprofit organizations — MDRC and Seedco — 
worked in close partnership with CEO to design the Family Rewards and Work Rewards 
demonstrations. MDRC is a nonpartisan, social policy research firm with extensive experience 
conducting large-scale demonstration projects using random assignment research designs to 
build rigorous evidence on what works to improve the well-being of low-income families. 


'Spark was evaluated by Harvard Education Labs, which developed the project in partnership with the 
New York City Department of Education. See Fryer (2010). 

4 The Opportunity NYC demonstration funders include the Bloomberg Philanthropies, The Rockefeller 
Foundation, The Starr Foundation, the Open Society Institute, the Robin Hood Foundation, the Tiger Founda- 
tion, The Annie E. Casey Foundation, American International Group, the John D. and Catherine T. MacArthur 
Foundation, New York Community Trust, and (for Spark only) the Broad Foundation. 
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Seedco works with local organizations to create economic opportunities for disadvantaged 
individuals and communities. The design team, made up of staff from CEO, MDRC, and 
Seedco, also conferred extensively with other New York City agencies and outside experts. 5 
Seedco, together with a small network of local, community-based organizations, operated the 
Family Rewards and Work Rewards programs, and MDRC is conducting the evaluations. 

In March 2010, MDRC published a report on the Family Rewards program’s early find- 
ings, covering the first one to two years of follow-up. 6 That period included the program’s start- 
up phase as well as a stage when the program was beginning to mature. Overall, that report 
showed that, while launching Family Rewards was challenging, the program passed a basic 
feasibility test: the operators successfully transferred substantial amounts of money to low- 
income families in a large American city, with all payments tied to a comprehensive set of 
conditions. The initial impact findings showed that the program began to achieve its primary 
short-term goal of reducing current poverty and material hardship. The program had also begun 
to generate some encouraging early effects on certain educational, health, and employment 
outcomes, but also left a number of important outcomes unchanged. The current report picks up 
the story, examining how the rewards were marketed and delivered during the third and final 
year of the program, how families’ experiences with the program evolved as it matured, how 
families experienced their exit from the program, and the program’s impacts through the early 
post-program period. 

In 2011, MDRC published a report that, using qualitative data collected through in- 
depth interviews with a sample of parents and children, explores the variety of ways in which 
parents and children communicated about the program’s educational incentives, the role of the 
program in parent-child relationships and family dynamics, and how the families used the 
reward monies they earned. 7 In 2012, MDRC published a report that uses data from a special 
survey of high school students and their parents to explore how the program affected the 
teenagers’ time use, spending patterns, parent-child relationships, and other outcomes. 8 Eater 
chapters in the current report draw on findings from these two special studies. 

The current report shows that the Family Rewards cash transfers continued to reduce 
poverty and important material hardships (such as an insufficient supply of food and a number 


5 Staff from the New York City Department of Health and Mental Hygiene, Department of Education, 
Human Resources Administration, Department of Consumer Affairs, and Department of Small Business 
Services were the main planning partners on Family Rewards, while the New York City Department of 
Housing Preservation and Development and the New York City Housing Authority were the main planning 
partners for the Work Rewards demonstration. 

6 See Riccio et al. (2010). 

7 See Greenberg, Dechausay, and Fraker (2011). 

8 See Morris, Aber, Wolf, and Berg (2012). 
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of housing-related hardships) throughout the program period. However, these effects began to 
attenuate after the rewards were no longer available to families. The program continued to 
produce a number of positive effects on some education, health-related, and employment 
outcomes, especially for certain subgroups, including positive impacts on graduation rates for 
ninth-graders who were better prepared for high school than their peers when they entered the 
study. But it still left other important outcomes unchanged. 

The evaluation of Family Rewards will continue through 2014, covering a follow-up 
period of up to six years after random assignment, depending on the data source. Thus, the 
findings presented in this report do not represent the final word on the program’s effectiveness. 


The Next Generation: Family Rewards and the 
Social Innovation Fund 

In 201 1, CEO and MDRC launched a new demonstration program to test a revised version of a 
CCT program — referred to here as “Family Rewards 2.0” — as a project of the federal Social 
Innovation Fund. 9 The early findings on the original Family Rewards program, presented in 
MDRC’s 2010 report on the program, 10 identified rewards and strategies that held some 
promise, as well as those that did not appear to work well. That evidence helped inform the 
design of the new program. The new model covers the same three domains (education, health, 
and workforce), but it has many fewer types of cash rewards, making communication and 
marketing to families easier. The revised model also calls for payments to be made on a more 
frequent basis and includes a new family guidance component. It is hoped that this “new and 
improved” version of Family Rewards, which is now being tested in the Bronx, New York, and 
in Memphis, Tennessee, in a new randomized control trial, will prove more effective than the 
original version. (See Chapter 7 for more information on the new model.) 

The remainder of this chapter reviews the Family Rewards demonstration, the pro- 
gram model, the characteristics of the study sample, and the overall approach for evaluating 
the intervention. Subsequent chapters present findings to date on the program’s implementa- 
tion experiences and impacts. For a detailed discussion of the origins of the demonstration, 


9 The Social Innovation Fund, enacted under the Edward M. Kennedy Serve America Act and adminis- 
tered by the Corporation for National and Community Service, targets public-private funds to expand effective 
solutions to pressing social problems through economic opportunity, healthy futures, youth development, and 
school support. This work seeks to create a catalog of proven approaches that can be replicated in communities 
across the country. See www.nationalservice.gov/about/programs/innovation.asp. 

10 See Riccio et al. (2010). 


4 



the theory of change underlying the model, and controversies over the approach, see MDRC’s 
2010 report. 11 


Program Overview 

Like other CCT programs in lower- and middle-income countries, the original Family 
Rewards program was a two-generation initiative with both shorter-term and longer-term 
poverty-reduction goals. 12 It included no new social services or case management. Instead, it 
attempted to use the offer of a new set of cash transfers in strategic ways to lessen immediate 
income-related hardships for poor families while simultaneously helping and encouraging 
those families to increase — or sustain — positive efforts to improve their own futures. The 
transfers were to function as an income supplement to improve families’ economic security in 
the shorter term, as enabling resources to help make short-term human capital investments 
feasible for them, and as inducements to encourage them to make those investments. Pay- 
ments, which were available for three years, were awarded when households met specific 
conditions in three key areas, or “domains”: 

• Education-based conditions, which include children’s superior attendance 
in school, meeting certain performance levels on standardized tests and other 
school outcomes, and parents’ engagement with their children’s education 

• Health-based conditions, which include maintaining health coverage for 
parents and their children, as well as age-appropriate preventive medical and 
dental checkups for each family member 

• Workforce-related conditions, aimed at parents, which include sus- 
taining full-time work and participating in approved education or job- 
training activities 

Family Rewards rested on the premise that financial incentives can influence individu- 
als’ short-term choices and actions in ways that can serve their best interests over the longer 
term. Economists, psychologists, and other scholars cite evidence, for example, that people 
often “discount the future,” meaning that they do not attach sufficient value to investments in 
education or health-related practices that can make them more economically secure or healthier 
because they do not fully recognize or appreciate the future payoffs that come from such 
investments. 13 In addition, for young children in particular, the future is very distant, and long- 


11 See Riccio et al. (2010). 

12 Riccio et al. (2010). 

13 See, for example, Fiszbein and Schady (2009) for a fuller discussion of this issue. 
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term rewards may be too abstract to be significant motivating forces. Community or peer group 
nonns, fed by observations of persistent intergenerational poverty, may also reinforce these 
perceptions, especially in high-poverty communities. At the same time, the simple lack of 
resources and other structural constraints among poor families can make it challenging to build 
human capital. For example, poor families may encounter difficulties getting access to good 
schools and enrichment programs for their children; paying for tutoring for children who need 
extra help; affording reliable child care when parents work; getting paid for time off from work 
to take their children to medical or dental checkups; finding dentists who are willing to take 
Medicaid; and even paying for transportation to and from low-wage jobs or job interviews, 
health visits, and school activities — in addition to the many additional impediments with 
which poor families must often contend. 

Financial incentives may help change the equation. In the face of more immediate and 
tangible rewards, people may take steps that serve their longer-term best interests, and perhaps 
even develop new habits, regardless of whether they fully recognize or believe in the longer- 
term value of those efforts. Furthermore, if the rewards are sizable, the extra resources can help 
make it more feasible for low-income people to undertake certain educational, health care, and 
work-related efforts in the short term. As resources accumulate from some activities, such as 
school attendance and doctor visits, they might help cover the costs of other activities and 
materials, such as the costs of educational materials or tutoring for children, transportation to a 
free dental clinic, the dental checkup itself (if not free or covered by insurance), clothes for a job 
interview, or tuition payments for a training program. In this sense, the conditioned rewards 
may function not only as financial inducements, but also as enabling resources. (See Figure 1.1 
for a depiction of the general theoretical model.) 

The Family Rewards model was operated in a total of six community districts — two 
each from the Bronx, Brooklyn, and Manhattan, as listed in Table 1.1. These six areas were 
chosen because they are among New York’s most persistently disadvantaged communities. At 
the same time, these communities are also diverse in terms of their racial and ethnic composi- 
tion (although most residents are black or Hispanic/Latino), and in terms of the prevalence of 
social service and health organizations. 14 

Family Rewards was targeted toward families in these six community districts who had 
incomes at or below 130 percent of the federal poverty level. 15 This standard is the same as the 
eligibility standard used for food stamps and a number of other benefit programs that serve very 


14 For more background on these communities, see Riccio et al. (2010). 

15 In the continental United States, the 2007 federal poverty level for a family of three was $17,170, and 
130 percent of the poverty level for such a family was $22,321. See U.S. Department of Health and Human 
Services (2007). 
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The Opportunity NYC Demonstration: Family Rewards 
Figure 1.1 

A Logic Model for Family Rewards 



NOTES: CCT = conditional cash transfer. Shaded boxes reflect the three program domains. 











The Opportunity NYC Demonstration: Family Rewards 

Table 1.1 

Key Neighborhoods in Selected Study Community Districts of New York City 


Borough and District 

Key Neighborhoods 

Bronx 

Community District 5 
Community District 6 

Morris Heights, University Heights, Fordham, Mount Hope 
East Tremont, Bathgate, Belmont, West Farms 

Brooklyn 

Community District 5 
Community District 16 

East New York, New Lots, Starrett City 
Brownsville, Ocean Hill 

Manhattan 

Community District 10 
Community District 1 1 

Central Harlem 
East Harlem 


low-income families, making it a widely accepted benchmark for identifying families in need of 
government cash transfer programs. As a two-generation intervention, the program was intend- 
ed for parents and their school-age children . 16 Furthermore, all parents and children had to be 
legal residents of the United States . 17 

Eligible families had to have at least one child in the fourth, seventh, or ninth grade. 
However, once a family enrolled in the program, all school-age or younger children were eligible 
to participate in it. The demonstration targeted students in these grades because they are at or near 
the start of critical educational transition years. For example, by fourth grade, children are making 
a transition from “learning to read” to “reading to learn” — in other words, applying their newly 
developed reading skills to acquire content knowledge. Without making that transition smoothly, 
future school work becomes more difficult, and children who fall seriously behind by the third or 
fourth grade tend to have difficulty catching up later . 18 The designers of Family Rewards thus 
hoped that offering three years of education incentives for the fourth-grade target group would 
help them do better in that critical year and successfully weather the transition to the first full year 

16 Adults were eligible only if they were the custodial parents or legal guardians of the eligible children, or 
a custodial parent’s cohabiting spouse or legally registered domestic partner. 

'’Undocumented residents were not included in the target population because they do not have a legal 
right to work in the United States. That means that they are not eligible for the program’s full-time work bonus. 
Consequently, including them would have impeded a full test of the two-generation CCT model in which 
workforce incentives for parents are an important program component. 

ls For one study showing this pattern, see Clotfelter, Ladd, and Vigdor (2006). 



of middle school, which typically begins in sixth grade in New York City. Similarly, rewards 
targeted to the cohort of seventh-graders were intended to help support and encourage those 
students to perform well in middle school through the end of ninth grade (the first year of high 
school), another critical transition stage. 

Although the incentives for the seventh-grade cohort ended after ninth grade, rewards 
for the ninth-grade cohort continued through the end of eleventh grade. Many high school 
students in low-income communities struggle in ninth grade. They begin to have more serious 
attendance problems and fall behind in credits, which puts them on a slippery slope toward 
dropping out of school. Students who succeed through the eleventh grade, however, are highly 
likely to graduate . 19 Thus, offering students education-focused incentives beginning in the ninth 
grade — often considered a “make-or-break” year — may help boost achievement at this very 
critical stage; continuing incentives for two more years may put them on a solid path toward 
graduation. 

Program Delivery Structure 

Seedco assembled a network of six local organizations in the designated community 
districts to assist in implementing Family Rewards . 20 Called Neighborhood Partner Organiza- 
tions (NPOs), these agencies recruited and enrolled eligible families into the research sample 
and served as the face of the program in the communities, providing ongoing customer service 
to participants who request assistance, such as in making claims for the rewards or for obtaining 
information about other services in the community. NPOs also conducted informational 
workshops on how to earn and claim rewards in each of the domains in which the incentives 
were offered. Seedco maintained a telephone helpline and a Web site to provide additional 
infonnation and assistance to families. 

Seedco verified that families earned rewards by using a combination of automated data 
from some City agencies and special “coupon book” forms submitted directly by participants. 
After verification, it initiated a process of transferring payments electronically into bank 
accounts that participants newly opened or into their existing accounts that they attached to the 
program, or, if they preferred, onto stored value cards (which are similar to debit cards but are 
not assigned to an individual account holder, such as prepaid telephone cards). The payments 
were made every two months (the length of a program “activity period”) and families could 
access the money at any time through any automatic teller machine (ATM). 

19 Kemple, Herlihy, and Smith (2005). 

20 These organizations are Urban Health Plan and BronxWorks (formerly Citizens Advice Bureau) in the 
Bronx; Brownsville Multi-Service Center and Groundwork, Inc., in Brooklyn; and Catholic Charities and 
Union Settlement Association in Manhattan. See Chapter 3 of Riccio et al. (2010) for a description of these 
agencies and the services they normally provide apart from Family Rewards. 
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By design, Family Rewards included no case management. This means that it made no 
provision for staff to develop action plans to address barriers in participants’ lives or to inter- 
vene in personal crises that may have made it difficult for them to succeed in the program. It 
also made no provision for staff to follow up with participants about their individual progress in 
meeting their goals or getting the services they needed, or to intervene directly with service 
providers on behalf of individual families, such as by helping to arrange tutoring for children, 
taking into account their children’s specific learning needs, or arranging child care that is 
appropriate to a family’s particular circumstances. Family Rewards also did not provide any 
direct services, such as tutoring, test preparation, job search classes, basic educational instruc- 
tion, or occupational training. 

The program’s designers excluded these forms of assistance and services for four main 
reasons. First, they wanted to test the power of the cash incentive alone. Second, in contrast to 
participants in CCT programs in poorer nations, they expected that many families would have 
access to services through other programs in the community, and that Family Rewards should 
not duplicate those services. Third, they reasoned that if the New York City model could 
succeed without those extra elements, it would be easier and less expensive to “scale up” — that 
is, to operate the program on a larger scale — as an ongoing policy. And fourth, they hoped that 
the program would be less burdensome to families if they were not required to have regular 
appointments with staff while also trying to balance work and family obligations. Thus, just as 
low-income workers need not take part in services in order to benefit from the Earned Income 
Tax Credit (EITC), 21 it was possible for Family Rewards participants to be “fully engaged” with 
the program without ever meeting with staff from Seedco and the NPOs (except to pick up their 
reward verification coupons). At the same time, the designers recognized that many families 
would need at least some guidance on where they could find the kinds of services and assistance 
that might enhance their success in the program — for example, where they could find tutoring 
or after-school programs, dental clinics, job search programs, and job training programs. For 
that reason, the designers included as part of the model an infonnation-and-referral component 
through which Seedco and the NPOs were expected to help educate families about relevant 
resources that are available in the community, such as by disseminating written resource guides 
listing agencies that might be appropriate for parents to contact, and by reminding families 
about those resources in optional workshops and in marketing materials. 


“'The EITC is a refundable federal income tax credit for low- and moderate-income working individuals 
and families, offered in part to offset the burden of Social Security taxes and to provide an incentive to work. 
See www.irs.gov. 
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The Family Rewards Incentives: Conditions and Reward 
Amounts 

During its first two years of operations, Family Rewards offered families an opportunity to 
receive 22 different types of cash rewards that could total several thousand dollars per year over 
a three-year period. As discussed shortly, the designers modified the rewards schedule for the 
third program year, dropping some rewards that appeared to be having little effect and simplify- 
ing several others. 

The total amount of money that families could receive through the program depended on 
the degree to which they met the qualifying conditions attached to the rewards and, importantly, 
on the number of children in the family, each of whom could be a source of substantial rewards. In 
developing the rewards schedule, the design team (CEO, MDRC, and Seedco) sought to provide 
families with many different ways to earn rewards. This was considered a way to achieve the 
program’s short-term goal of immediate poverty reduction while, at the same time, not attaching 
excessive amounts of money to any given reward. A broad reward structure that included a mix of 
easier-to-eam and harder-to-eam rewards would allow most families to earn some rewards. In 
addition, the designers had little evidence to decide a priori which rewards would be the most 
productive in changing family outcomes. 22 However, with 22 different rewards (initially), the 
program was more complex than was desirable. That complexity meant that the program staff 
would need to make more intensive efforts to educate families about the full incentives offer and 
that families would undoubtedly take more time to respond to the full range of rewards than 
would be true in a simpler program. 

The design team faced another important constraint in trying to ensure that rewards 
would be paid close to the time when families met the required conditions. Because of the large 
number and diverse types of rewards, the process of verifying compliance was complicated, and 
for practical reasons it was decided that payments would be made only every two months. There 
was also no way to avoid the long natural lags between activities and payments for certain 
rewards that were based on administrative data for verification. For example, it took time to 
distribute rewards for achievement on standardized academic tests because the New York City 
and New York State education departments took several months to score the tests and provide 
that information to the program. 

Table 1.2 summarizes the incentives schedule, showing the specific types of behavior 
and achievements that earned payments and the amounts of those payments. In general, the 
payment amounts for the education and health components involved larger sums than has 
been true of most prior tests of incentives strategies in those two fields. In contrast, the full- 


22 For a fuller discussion of considerations that influenced the program design, see Riccio et al. (2010). 
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The Opportunity NYC Demonstration: Family Rewards 

Table 1.2 

Schedule of Incentive Payments, by Domain 


Domain 

Schedule of Incentive Payments 

Children’s educational efforts and 
achievement 


Grades 1-8 (payments made to parents) 
Attendance 

$25 per child per month (maximum: $25 per month of 
school year) for superior attendance (95% of scheduled 
days, with provision for extended illness). Discontinued 
for Year 3. 

Parent-teacher meetings 

$25 per meeting, twice per year (maximum: $50 per 
child per year) for parent’s attendance at parent-teacher 
conferences. Modified for Year 3 to include other 
parent-teacher consultations. 

Library card 

$50 paid once during program per child for having a 
public library card. Discontinued for Year 3. 

Reviewing results of low-stakes 
interim tests 

$25 for parents to acquire and review on their own their 
children’s performance on interim standardized tests 
intended to help teachers diagnose students’ progress 
(up to 5 times per year; maximum: $125 per child per 
year). Discontinued for Year 2. 

Test scores (starting in grade 3) 
For grades 3-5 

$300 per child for scoring at a level 3 (indicating 
proficiency) or above on standardized ELA test, or 
(starting in grade 4) for improving by at least 1 level 
over prior year’s level; same for standardized math test. 
(Maximum: $600 per child per year.) 

For grades 6-8 

$350 per test for meeting the same conditions as for 
grades 3-5. (Maximum: $700 per child per year.) 

Discussing results of annual ELA and 
math tests with school (starting in 
grade 3) 

$25 per test, once per year (maximum: $50 per child per 
year) for parents to discuss child’s test results with 
teachers or principal. Discontinued for Year 3; incorpo- 
rated into parent-teacher meeting reward. 


(continued) 
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Table 1.2 (continued) 


Domain 

Schedule of Incentive Payments 

Grades 9-12 (payments split between 
parents and students, as indicated) 


Attendance 

$50 per child per school month (maximum: $500 per 
year) for superior attendance (95% of scheduled days, 
with provision for extended illness). (50% paid to 
student, 50% paid to parent.) 

Parent-teacher meetings 

$25 per meeting, twice per year (maximum: $50 per 
child per year), for parent’s attendance at parent- 
teacher conferences. (100% paid to parent.) Modified 
for Year 3 to include other parent-teacher consulta- 
tions. 

Library card 

$50 paid once during program per child for having a 
public library card. (100% paid to student.) Discontinued 
for Year 3. 

Test scores 

$600 per child for passing (scoring 65 or above on) each of 
5 Regents tests (maximum: $3,000 during program). 3 (100% 
paid to student.) 

Credit accumulation 

$600 per child per year for accumulating 1 1 credits during a 
school year. (50% paid to student, 50% paid to parent.) 

PSAT 

$50 for taking PSAT test up to 2 times (maximum: $100 
during program). (100% paid to student.) 

Graduation 

$400 payment for graduating from high school. (50% paid 
to student, 50% paid to parent.) 

Family preventive health care 
practices 


Maintaining health insurance 

$20 per month (maximum: $240 per year) for each parent who 
maintains public health insurance (including Medicaid and 
Family Flealth Plus coverage). Discontinued for Year 3. 


$20 per month (maximum: $240 per year) for maintaining 
Medicaid or CHIP coverage for all children (together). b 
(Not for TANF recipients due to near-automatic Medicaid 
enrollment.) Discontinued for Year 3. 


$50 per month (maximum: $600 per year) for each parent 
who maintains private/employer health insurance. $50 per 
month (maximum: $600 per year) for maintaining pri- 
vate/employer insurance for all children (together). b 
Discontinued for Year 3. 


(continued) 
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Table 1.2 (continued) 


Domain 

Schedule of Incentive Payments 

Nonemergency health screenings 
and early intervention 

$200 per family member per year for completing an annual 
nonemergency medical checkup. Physician must fill out 
“preventive health care form’’ indicating that a minimum set of 
age-appropriate screenings and assessments was conducted 
and that other health information was reviewed with the 
patient and/or parent. 


$100 per family member per year for completing a physician- 
advised follow-up visit within a specified time frame. Discon- 
tinued for Year 3. 


For young infants and toddlers (children under 30 months of 
age): $200 per child for completing a pediatrician-advised 
early-intervention evaluation. 


Dental care: $100 per family member for cleaning and checkup; 
once per year for ages 1-5 and twice per year for ages 6 and older. 

Adult workforce efforts 


Sustained full-time employment 

$300 for working full time (an average of 30 hours per week 
for 6 weeks or more in each 2 -month payment period — that 
is, approximately 75% of the time) (maximum: $1,800 per 
year at $150 per month). 

Education and training while 
employed 

Payments for completing an approved education or training 
course while holding a job. Must work at least 10 hours per 
week while attending course. ( Work requirement discontinued 
for Year 3.) $300 per each course lasting 35-70 hours; 0 $400 
per each course lasting 71-140 hours; $600 for each 141 -hour 
increment of a course lasting at least 141 hours (maximum: 
$3,000 per adult during program). (Training may include ESL, 
basic skills, and GED courses.) 


NOTES: ELA = English language arts. ESL = English as a Second Language. GED = General Educational 
Development. 

a High school students (grades 9-12) were eligible to earn rewards for the following Regents exams: 
English, one of any math exams (including Math A, Math B, Integrated Algebra, Geometry, and Algebra 
2/Trigonometry), U.S. History and Government, Global History and Geography, and one of any science 
exams (including Living Environment, Chemistry, Physics, and Earth Science). 

b Seedco experienced logistical difficulties in using Human Resources Administration data to find out 
whether all children in a household were covered by public insurance in a timely manner. As a result, 
families were rewarded for public health insurance coverage if any child in the household was covered, in 
practice. As it turned out, most families either had health insurance coverage for all of their children or did 
not have it for any. 

°The 2010 report incorrectly stated this reward amount as $200. 
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time work bonus was somewhat smaller than similar incentives tried in demonstration projects 
that have tested or are currently testing wage supplementation strategies, but they were still 
substantial. When combined, the full set of cash transfers represents a potentially very signifi- 
cant increase in the incomes of the very poor families, allowing them to receive several thou- 
sands of dollars in extra income per year. The cash transfers did not affect eligibility or payment 
amounts for most existing government transfer benefits, including Temporary Assistance for 
Needy Families (TANF), food stamps, 23 Medicaid, the Children’s Health Insurance Program 
(CHIP), housing assistance, or the EITC. 24 This strategy avoided undercutting the value of the 
incentives offer and greatly simplified the implementation of the program. 

Education Rewards 

The first panel of Table 1.2 presents the schedule of payments pertaining to children’s 
education. For all students, rewards during the first two program years were attached to 
superior school attendance, obtaining a library card, and parents’ participation in parent- 
teacher conferences. 

For students in elementary and middle school (starting with children in third grade, 
when standardized testing begins in New York City schools), additional rewards were offered 
for scoring at or above a threshold level on annual standardized tests in English language arts 
(ELA) and math. 25 Payments were made if a student achieved a score that fell within the 
“proficient” range (a level 3 or 4 on a four-level proficiency scale) or improved his or her score 
in a given year over the prior year by at least one full level on that four-level scale. 26 The hope 
was that these rewards would encourage parents to engage more deeply with their children on 
activities that, in general, might help improve their perfonnance in school in ways that would 
eventually be reflected in their standardized test scores, such as by monitoring their homework 
more closely, talking with them more about how they could perform better in school, taking 
advantage of other programs and school resources in the community that might support their 
perfonnance, and talking more with their teachers. As a more direct incentive for increased 


23 The federal Food Stamp program was renamed the Supplemental Nutrition Assistance Program (SNAP), 
beginning on October 1, 2008. However, this report refers to the program as the Food Stamp program because 
that is the name by which it is more commonly known at this time. 

24 However, the CCT payments may affect the Supplemental Security Income (SSI) payments of partici- 
pating families who were contending with physical or mental health disabilities and receiving such benefits. A 
waiver had been sought from the Social Security Administration but the request was not approved. 

25 Because standardized testing begins in third grade, the opportunity to earn a reward through improve- 
ment from the prior year did not apply. It also does not apply to students who are new to New York City 
schools and have no prior-year standardized test score. 

26 In 2010, New York State education officials increased the threshold scores that students had to meet on 
standardized tests, because they considered the existing thresholds too easy to reach and, therefore, not a good 
indicator of performance that met grade-level standards. 
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interaction with the schools, the program offered an additional reward for parents to discuss the 
results of the annual tests with their children’s teachers or school principals. (Most other 
countries’ CCT programs do not attach rewards to achievement outcomes.) 

For high school students, Regents exams are the relevant tests. These exams are admin- 
istered statewide each year by the New York State Board of Regents, an entity that sets standards 
and regulations that apply to all public schools and universities in the state. In order to graduate 
from high school with a diploma that is recognized by the Board of Regents, students must pass 
at least five tests in the following subject areas at some point during their high school career: 
English, mathematics, global history and geography, U.S. history and government, and science. 
Family Rewards offers students a separate payment for passing each of these five main content- 
area exams (that is, obtaining a score of at least 65). 27 High school students could also earn a 
small additional payment for taking up to two Preliminary Scholastic Aptitude Tests (PSATs), 
which they could do without charge in New York City. (These tests are designed to help students 
prepare for the Scholastic Aptitude Tests, or SATs, that many colleges and universities require 
for admission.) Additional payments were attached to earning a minimally acceptable number of 
credits in a given academic year (1 1 each year toward a total of 44 credits needed to graduate). 
This reward was created to encourage students to pass all their courses each academic year 
(taking advantage of summer school opportunities if necessary) so that they would remain on 
track to graduate in four years. An additional reward was available for graduating. 

Preventive Health Care Rewards 

The second panel of Table 1.2 (“Family preventive health care practices”) shows the 
preventive health care rewards. These rewards covered activities pertaining to health insurance, 
preventive health care checkups, and dental care. 

Many families who enrolled in the study were eligible for publicly provided health in- 
surance through Medicaid, or in some cases through CHIP or Family Health Plus (FHP) in New 
York State. 28 However, families must comply with annual recertification requirements for these 


27 Special education students with Individualized Education Plans (IEPs) — mandated by the Individuals 
with Disabilities Education Act to help children with disabilities achieve their educational goals — need to pass 
the Regents Competency Test, a modified version of the Regents exam. 

28 Medicaid is available to pregnant women and children under age 6 whose family income is at or 
below 133 percent of the federal poverty level, for children ages 6 to 19 with family income up to 100 
percent of the federal poverty level, and for families receiving government income support through the 
federal TANF program or New York State’s Safety Net Assistance program. CHIP (known as Child 
Health Plus in New York State) is a federal public health insurance program, administered by states, for 
families with children who have incomes that are too high to qualify for Medicaid but are within 200 
percent of the federal poverty level. FHP is a New York State public health insurance program for adults 

(continued) 
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programs, and many eligible families fail to complete the process. Consequently, “churning” on 
the rolls can be significant, with many otherwise eligible families losing their coverage . 29 
Family Rewards incentive payments for maintaining health insurance were offered to address 
this problem, encouraging families to keep their coverage in place. The program also offered 
payments to families in which the parent was not eligible for public health insurance but had 
access to private, employer-sponsored health insurance. Some families may forgo that insurance 
because of copayment costs for premiums or services; in these cases, the rewards, which were 
set at a higher amount than for maintaining public insurance, were intended to encourage 
families to make those copayments and get the insurance. Participants who were receiving 
TANF or Safety Net Assistance (SNA) benefits were not eligible for the program’s health 
insurance rewards . 30 This was because families who were receiving those benefits were routine- 
ly enrolled in Medicaid at the time of application and did not have to reestablish their eligibility 
as long as they remained on TANF or SNA. 

The second set of health care incentives was designed to encourage families to get 
comprehensive, nonemergency physical examinations — the cornerstone of good preventive 
health care practice. The main objective was to ensure that family members got comprehensive 
medical assessments that could lead to the early diagnosis and treatment of health problems that 
might become more serious over time, and that could also alert them to or reinforce the im- 
portance of healthy lifestyle choices (for example, healthful eating, exercise, and protection 
against sexually transmitted diseases). At the same time, it was hoped that by promoting 
preventive health care visits, families would be more likely to establish a “medical home” — 
that is, a relationship with a regular doctor (or health care institution) who would maintain their 
medical records and understand their medical history, and to whom they could turn when 
problems arose, rather than resorting first to hospital emergency rooms. 

To encourage thorough exams during these visits, the program designers, in consulta- 
tion with the New York City Department of Health and Mental Hygiene, created a special 
“preventive care checklist form” that identified a set of common health conditions that doctors 
are expected to explore or screen for in any thorough annual physical examination. This form, 
which had to be signed by the participant’s doctor in order to verily that the activity had been 


who are age 19 to 64 and who have income or resources too high to qualify for Medicaid; those with 
children can qualify with family incomes up to 150 percent of the federal poverty level. 

~ 9 One unofficial estimate that the New York City Human Resources Administration provided during the 
Family Rewards design phase suggested that 3 1 percent of Medicaid cases did not complete a recertification 
annually and were closed, although 27 percent of those closed cases were reopened within nine months. 

30 Safety Net Assistance is a New Y ork State welfare program for various low-income populations. For 
income-eligible families with dependent children, it allows those who have exhausted their five-year eligibility 
for cash assistance under TANF to continue receiving cash assistance on similar terms and conditions that 
applied under TANF, but paid out of non-federal funds. 
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completed, was tailored to the different needs of adults, teenagers, and younger children. For 
infants and toddlers, the form included a standard set of questions to encourage the doctor to 
screen for developmental problems and to make an appropriate referral for a fuller early 
intervention evaluation when warranted. An additional payment was offered to parents to 
encourage them to follow through with such a comprehensive evaluation (which is free in New 
York City) if their children’s pediatricians advised them to do so. 

The design team sought to create incentives for getting follow-up care that would make 
it practical for those who needed such care to get it, but without encouraging those who did not 
need care to try to get it just to earn the extra rewards, which would waste medical resources 
and program dollars. The decision was made to attach an incentive payment for one follow-up 
visit per family member per year in cases where the doctor indicated the need for it on the initial 
health care checklist (and specified the purpose and time frame for making the follow-up visit 
on a subsequent fonn). The payment amount for the second visit was set at half that of the first 
visit, to help temper the incentive to seek unneeded medical care. 

Finally, the health component included incentives for preventive dental care (regular 
cleanings and checkups). Although many dentists do not accept Medicaid, a number of dental 
clinics around the city offer free or reduced-cost dental care. Identifying them, getting to 
them, and paying them (if they charge a fee) were expected to be significant burdens for low- 
income families. The incentive payments were intended to help compensate for those extra 
burdens and costs. 

Workforce Rewards 

The third panel of Table 1.2 (“Adult workforce efforts”) presents the workforce com- 
ponent, which was aimed at the parents and had two main features. The first was a payment for 
sustained full-time employment. Operationally, this meant that a participating parent had to 
work at least 30 hours per week for six weeks out of every eight- week activity period. Allowing 
for some “downtime” was a way of recognizing that, for many low-wage workers, job turnover 
is common, sometimes because the job itself ends. Those who were in this situation or who left 
work for other reasons would have a strong incentive to seek another full-time job quickly. 31 

The workforce incentives also incorporated payments for completing approved educa- 
tion and training activities that could help build parents’ human capital so that they could 
qualify for higher-skilled and better-paying jobs. The courses could be shorter term or longer 
term, and the incentive payments were tailored with that in mind, providing a higher payment 


3 1 This reward is modeled after a bonus that was tested in an employment retention and advancement 
demonstration project in the United Kingdom; see Riccio et al. (2008). 
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for a longer-term course. Instruction could include not only specific occupational skills training, 
but also instruction in English as a Second Language (ESL), adult basic education (ABE), and 
General Educational Development (GED) preparation. 32 To discourage participants from 
dropping out of the labor force in order to undergo training, which would be inconsistent with 
New York City’s welfare-to-work policies, the program’s designers further required that the 
training reward be available only to parents who were working at least 10 hours per week. 

Year 3 Modifications 

The program’s designers recognized early on that the incentives described above might 
need to be modified as the program unfolded. They knew before the program was rolled out that 
their estimates of how many families would earn rewards, and, hence, the cost of those incentives, 
were largely guesses. Furthermore, in the absence of a pre-study operational pilot, they could not 
fully anticipate all the practical difficulties that would be involved in marketing, verifying, and 
processing the long list of incentives to be offered. However, a better understanding began to 
emerge as operational experience grew and some preliminary impact findings became available. 
In the summer of 2009, as the end of the second program year was approaching, CEO, Seedco, 
and MDRC agreed on a set of modifications to the incentives schedule for the program’s third and 
final year that would simplify the program and reduce its overall costs, both of which would make 
the program more feasible to replicate if successful. These modifications included the following: 

• Children’s education. First, the attendance reward was discontinued for el- 
ementary and middle school students because average attendance rates 
proved to be very high already for those grade levels, as evidenced by the 
control group’s patterns, and it appeared that the program was producing lit- 
tle further gain. Second, the reward for parents to discuss their children’s an- 
nual ELA and math test results with teachers was discontinued as a separate 
reward, and the parent-teacher conference rewards were no longer limited to 
attendance at the official semi-annual parent-teacher nights sponsored by the 
schools. Instead, a single, consolidated reward for parent-teacher exchanges 
was created and offered to parents twice a year (once during the first half of 
the year and once during the second half), to encourage them to talk with 
teachers about test scores and any other issues concerning their children’s 
school performance, and at times of the parents’ choosing. Third, payments 


32 In order to earn the payments for these activities, participants in ABE, GED preparation, or ESL classes, 
for which standards of completion are often ambiguous and compliance hard to measure, had to provide 
documented evidence from their providers indicating that they made satisfactory progress in their classes and 
that they participated for the required number of hours established by Family Rewards for a given level of 
payment. 
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for library cards were discontinued because most children who were likely to 
get them were believed to have done so within the first two years. 

• Family health care. First, all health insurance rewards, a costly component, 
were discontinued for Year 3. This was done partly because preliminary data 
from the evaluation suggested that insurance coverage rates were already 
very high for adults and children in the study. Second, rewards for doctor- 
recommended follow-up visits were discontinued because of the complexity 
of distinguishing visits that were truly separate follow-up visits. 

• Parents’ work and training. The requirement that parents must be working 
at least 10 hours per week while in education or training programs in order to 
earn that workforce reward was eliminated. Very few adults received that 
reward during the first two years, a problem that may have been exacerbated 
by the poor economy, which made it more difficult to find work. 

Family Rewards Compared with Other Conditional Cash Transfer 
Programs 

Family Rewards shared important principles with CCT programs in other counfries, particularly 
its dual emphasis on immediate hardship reduction and human capital development. However, 
CCT programs in most other countries are at the heart of their social protection systems and are 
the main source of government cash assistance. In contrast, Family Rewards was layered on top 
of an already well-developed safety net in New York City. It offered families a chance to secure 
extra income. Indeed, as described later in this chapter, many of the families in the program 
were already receiving food assistance, public health insurance, cash welfare, and/or rent 
subsidies. New York also includes a broad network of social services programs. Thus, Family 
Rewards had to be adapted to a context in which the underlying social protection system is very 
different from that available in other countries. 

Family Rewards shared with other CCT programs a central focus on children’s educa- 
tion and family preventive health care. However, the reasons children may not remain in and 
progress in school or get the preventive care they need are also different from those found in 
Mexico and elsewhere, and the reward structure reflected those differences. More generally, 
New York’s program was distinguished by the sheer number of rewards it offered (22 separate 
ones initially). It was also unusual in including rewards for educational achievement, not just 
attendance, and for work and training. In further contrast to most other CCT programs, Family 
Rewards was a short-term, time-limited intervention and was delivered entirely by private, 
nonprofit organizations rather than by government institutions. As an entirely urban program, it 
operated in a very different social and economic environment than the more heavily rural CCT 
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programs in the rest of the world. And, of course, as a pilot project, it operated on a much 
smaller scale than most other CCTs. 


Evaluation Overview and Data Sources 

When completed, the Family Rewards evaluation will show whether the program has the 
positive effects that its designers hope for. This report shows the effects, or “impacts,” of the 
program on family outcomes measured three to four years after participants entered the study, 
depending on the data source. This time period makes it possible to assess the program’s effects 
during the full three years in which the incentives were offered to families as well as (for some 
measures) during the beginning of the post-program period, after the program and the incentives 
ended. The post-program period is especially important for determining whether any positive 
impacts that were achieved while the incentives were available are sustained or grow, and 
whether negative effects on education or other outcomes emerge after the incentives are no 
longer offered. (The final report will cover a five- to six-year follow-up period.) 

The evaluation is using an extensive set of quantitative and qualitative data. This infor- 
mation includes administrative records on school outcomes, employment, earnings, public 
health insurance, and welfare and food stamp payments obtained from various New York City 
and New York State agencies; two waves of a survey in which a subset of parents in the 
program and control groups are interviewed (this report focuses on the second wave of the 
survey administered at about 42 months after families entered the study); 33 program-related data 
on reward payments obtained from Seedco; and qualitative data obtained through in-depth 
interviews with a sample of program participants and through observations of staff carrying out 
program activities at Seedco and the NPOs. (See Appendix Table A.l for the specific calendar 
periods covered by each data source.) 


Who Enrolled in the Family Rewards Sample? 

The six NPOs began recruiting the sample for Family Rewards in July 2007, working from lists 
of potentially eligible families constructed with data obtained from the New York City Depart- 
ment of Education. The lists contained the names and contact information for families who 
lived in the targeted areas, were enrolled in the National School Lunch program (which was 
open to families with incomes at or below 130 percent of the federal poverty level), and had 
children who were expected to enter the targeted grades in September 2007. The NPO staff 


33 The survey had an overall response rate of 79 percent, with similar results for the program and control 
groups. For further details on the survey and response bias analysis, see Appendix J in the supplement to this 
report, Riccio et al. (2013), which is available at www.mdrc.org. 
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succeeded in enrolling approximately 4,800 families (and more than 11,000 children) in the 
Family Rewards sample in the six-month period between July and December 2007. 34 

Appendix Tables A.2 to A.4 show the characteristics of all families, parents, and chil- 
dren in the sample at the time of random assignment, combining program and control group 
members from all the NPOs. The recruitment methods that NPOs used called for staff to make 
repeated attempts to contact hard-to-reach families, rather than filling up the sample with more 
easily located families or those who were most eager to join. This was done in order to try to 
generate a sample that would reflect that broader target population from which it was drawn. A 
comparative analysis of background characteristics suggests that the enrolled sample is a 
reasonable approximation of the larger targeted population. For example, the parents who were 
enrolled in the Family Rewards sample share many of the same characteristics as other low- 
income parents living in their communities. Moreover, the pre-program average test scores of 
the children who were enrolled in Family Rewards were virtually the same as the test scores of 
children on the sample recruitment list who did not enroll in the study. 35 

Family and Parent Characteristics 

As shown in Appendix Table A.2, most of the families who were enrolled in the Family 
Rewards sample were one-parent families (81 percent). Over half (57 percent) had one or two 
children, while 43 percent had three or more. Most families were receiving some fonn of 
government assistance. For example, 53 percent were receiving housing assistance in the fonn of 
either public housing (30 percent) or Section 8 rent vouchers (23 percent); over 59 percent were 
receiving food stamps; and 24 percent were receiving TANF or SNA. In fact, only 13 percent of 
enrolled families were not receiving any of these public benefits or housing assistance. 

Appendix Table A.3 shows the background characteristics of the parents in the study 
sample. On average, parents were 40 years of age. Nearly all were either black, non- 
Hispanic/Latino (51 percent) or Hispanic/Latino (47 percent). Educationally, they were a 
diverse group. While 40 percent did not have a GED certificate or a high school diploma, 14 
percent had either an associate’s or bachelor’s degree. Their employment status also varied 
widely, with slightly more than half (53 percent) holding a job. Overall, 40 percent of parents 
were working full time (at least 30 hours per week), and 13 percent were working part time (not 
shown in table). 


34 See Chapter 2 in Riccio et al. (2010) for full details on the sample recruitment and enrollment process. 
For a comparison of the baseline characteristics of program and control group sample members, see Appendix 
A of that report. 

35 See Chapter 2 in Riccio et al. (2010). 
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Children’s Characteristics 


Appendix Table A.4 shows that 94 percent of enrolled children were bom in the United 
States. The vast majority of children were attending public schools (98 percent), with 51 percent 
entering the targeted grades: 17 percent were entering fourth grade, 16 percent were entering 
seventh grade, and 19 percent were entering ninth grade. The remaining children were siblings 
of those targeted students and were enrolled in other grades. In the previous school year, 14 
percent of all students had been enrolled in special education and 13 percent in the English 
Language Learner (ELL) program. 36 The children were spread across 407 elementary schools, 
358 middle schools, and 390 high schools throughout the city, or 1,155 schools in all (not 
shown in table). 

Most of the children’s parents reported attending a parent-teacher conference in the last 
year. Only 5 percent of children had a parent who reported never attending a parent-teacher 
conference. An additional 35 percent of children had a parent who reported attending one or 
two parent-teacher conferences. Because Family Rewards includes payments for attending 
parent-teacher conferences, these high rates at random assignment suggested that many families 
might be eligible for the payment. 

Health Characteristics 

The responses to health-related questions were similar for parents and children (see Ap- 
pendix Tables A.3 and A.4). Both children and parents had high rates of health insurance 
coverage and annual medical exams, as discussed further in Chapter 5. Only 6 percent of 
parents and 3 percent of children had no health insurance coverage. Public health insurance was 
the most common form of coverage, with 7 1 percent of parents and 8 1 percent of children 
having such insurance. The rates of self-reported medical checkups were high. For example, 82 
percent of parents reported having an annual medical exam within the past year, and this was 
also the case for 91 percent of the children in the sample. As might be expected, fewer partici- 
pants reported having an annual dental checkup at the time of random assignment, with 65 
percent of parents reporting a dental checkup in the past year, along with 75 percent of children. 


Structure of This Report 

This report updates findings on the wide range of topics that were explored at an early stage of 
the program in the evaluation’s first report, published in 20 10. 37 Chapter 2 begins the analysis 

36 English Language Learners are children who speak a language other than English at home and score 
below proficient on English assessments when they enter the New York City school system. See 
http://schools.nyc.gov/Academics/ELL/default.htm. 

37 See Riccio et al. (2010). 
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by examining the operation of Family Rewards as it matured, through the third and final year of 
the program. It also explores how families’ understanding of the incentives offer evolved over 
time, the number and amounts of rewards they earned, and their perspectives as they adjusted to 
the absence of reward payments after the program ended. Chapter 3 reports on Family Rewards’ 
impacts on a variety of outcomes that are most closely associated with the program’s short-term 
poverty-reduction goals. In addition to measuring impacts on income and poverty, that chapter 
examines the program’s effects on commonly used indicators of material hardship, economic 
well-being, financial and banking behaviors, and asset building. Importantly, it considers how 
the effects on those measures changed after the special cash transfers were no longer available. 
Chapters 4 through 6 examine, by domain, the extent to which program participants earned 
particular types of rewards, and the program’s impacts on children’s education, family health- 
related outcomes, and parents’ workforce outcomes. Chapter 7 offers concluding thoughts on 
the current analysis, describes what is ahead in the final report (scheduled for 2014), and 
describes the new version of Family Rewards that is now being tested in a companion study in 
the Bronx, New York, and Memphis, Tennessee. 
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Chapter 2 

Operating Family Rewards and Patterns of 
Reward Receipt 


Family Rewards aimed to encourage families to engage in behaviors that would enhance their 
“human capital” by offering financial incentives to participants for completing activities in the 
domains of education, health, and work. Enrolled elementary, middle, and high school students 
were eligible to earn the education rewards; every enrolled family member (including children 
who were not yet school age) could earn the health rewards; and enrolled adults could earn the 
workforce rewards. The program operated from September 2007 to August 2010 and served 
nearly 2,400 families in six of the highest-poverty communities in New York City. 

Many of the key implementation questions surrounding Family Rewards were ad- 
dressed in detail in the first report about the program. 1 That report explained how the functions 
that were required to operate a conditional cash transfer (CCT) program were shared by several 
organizations. Seedco, the lead implementing organization, developed systems to verily earned 
rewards and make payments to families, as well as to resolve payment issues. Seedco also 
provided general guidance and technical assistance to six Neighborhood Partner Organizations 
(NPOs), whose staff marketed the rewards to families, provided in-person assistance with 
claiming rewards, and made referrals to existing community agencies that could help families 
achieve the conditions for earning rewards. MDRC’s early assessment of the implementation of 
the program model found that, although the program staff struggled in the first year to balance 
the demands of recruiting the sample and providing customer service supports to program group 
members, they were operating the program as designed and using creative strategies to market 
the incentives to participants by the second year. Participants rated the customer service support 
very positively on an 18-month survey that they were asked to complete. Engagement with the 
program, as defined by the regularity with which families earned cash rewards and the total 
amount of the average reward, was also very high. 

This chapter does not duplicate the descriptive data on program operations that were 
presented in MDRC’s earlier report. Instead, the goal is to take advantage of having three full 
years of program data to summarize families’ overall experiences, track trends in earning 
rewards, and highlight new dimensions of that experience, especially with respect to high 
school students, while drawing attention to the distinctive operational issues that were associat- 
ed with the final year of the program. The chapter addresses these main questions: Were 


'See Riccio et al. (2010). 
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families able to understand and navigate this type of incentives-only CCT program? Which 
types of families were most successful in earning rewards and why? Were families prepared for 
the end of the financial incentives, and did they make provisions to sustain any positive behav- 
ioral, motivational, or material changes that they had experienced during the program? 

In the following sections, this chapter describes two key changes that occurred in the 
third year of the program and how they affected the earning of rewards. Because it was the 
final year of the program, Seedco and the NPOs used the “last chance” message to motivate 
families to earn rewards and created a set of exit materials to prepare families for the potential 
income “cliff.” At the same time, the program eliminated several types of rewards that had 
been available in prior years. Thus, for families to maintain the same level of earning as in 
previous years, they would have had to earn rewards more frequently or in more categories 
than they had before. The data on reward earnings show that, although earnings from the 
program continued to be strong, families, on average, did not appear to adopt successful 
strategies to maximize earnings in the third year. 

A deeper analysis of the pattern of earnings shows which types of families earned the 
most and the least from the program. The 2010 report found that the families who earned the 
most rewards were larger, had more children in high school, and were less disadvantaged at the 
time of random assignment. Language and race were not associated with the level of rewards 
earned, but immigrant families tended to be among the highest earners. This general pattern 
holds when all years of program earnings are combined. 

Finally, this chapter investigates the uses that families made of their earnings at the end 
of the program and in the months directly after they received their final payments. Parents were 
asked on a 42 -month survey whether they planned to make up for the loss of their Family 
Rewards earnings, and, if so, how. Given that the program did not offer individualized counsel- 
ing about next steps, participants’ responses to this question reflect aspirations about what they 
believed they could accomplish on their own. Nevertheless, it is important to understand 
participants’ intentions for coping with the loss of the rewards — both as a source of income 
and as a means for parents to motivate children — in order to provide some indication of the 
possible long-term influences of this program on economic stability and family dynamics. 

Data for the new analyses presented in this chapter come from a special section of the 
42 -month parent survey, field observations, and two waves of in-depth qualitative interviews. 
For the in-depth interviews, 50 parents and 39 children from the same families were first 
interviewed between October 2009 and April 2010. Of this sample, 22 parents and 14 high 
school students were interviewed in-depth again a few months after the program ended (in early 
201 1). Reference is also made, where appropriate, to data that were collected on the 18-month 
parent survey. 
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Three Years of Family Rewards 

Given the number and diversity of rewards that the program offered, setting up a system to keep 
track of all of them, to determine when families met the conditions for earning them, and to 
make timely payments to families was a major implementation challenge. Activities and 
accomplishments were divided into those that required the family to submit a “coupon” in order 
to receive payment and those that were verified through administrative records. Most of the 
education rewards and the public health insurance rewards were verified through administrative 
data supplied directly to Seedco by the New York City Department of Education and the New 
York City Human Resources Administration. These are referred to as “automatically verified” 
rewards, and they required no effort on the part of families to claim payments. Administrative 
records were not available for any of the other rewards; thus, for those, families had to verily 
their compliance and submit claims manually using special coupons that were created for the 
program. Coupons were double-sided, colorfully designed sheets of paper that listed the name 
of the activity, some identifying information about the participant, the dollar value for complet- 
ing the activity, and instructions for submitting it to Seedco for payment. The coupon itself 
either had to be signed by a teacher or doctor to verify completion of an education or health 
activity, or the participant had to photocopy and attach a prescribed document to it as a form of 
proof, such as a copy of a library card or pay stub. At the beginning of each program year, the 
NPOs invited families to pick up a personalized binder prepared by Seedco containing the suite 
of coupons for which their enrolled family members were eligible. Families were provided with 
self-addressed, stamped envelopes for mailing coupons. 

Families were paid for their completed activities every other month. The 12-month aca- 
demic year was divided into six 2-month “activity periods.” Participants completed the reward- 
ed activities during a 2-month period, submitted any coupons that were required to verify 
completion of those activities by the fifteenth day of the following month, and received pay- 
ment on the fifteenth day of the subsequent month. For example, participants who completed 
activities in September and October submitted any necessary paperwork by November 15 and 
were paid on December 15. This means that while participants were submitting coupons and 
awaiting payment from the first payment period, they had already entered the next one. In the 
example above, participants could have been completing activities in November and December 
in order to submit any required coupons by January 15 and receive payment on February 15. 

When families were initially oriented into Family Rewards, they were encouraged with 
a $50 incentive payment to submit a bank account number to the program to receive reward 
payments by direct deposit. Having linked bank accounts fulfilled a key operational requirement 
by enabling regular transfers of money to participants, but it was also seen by the program 
designers as a way to connect a low-income population to the mainstream banking system. 
Families could use their existing bank accounts or sign up for “Opportunity NYC accounts.” 
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These special “safe” accounts were created through negotiations between the City’s Office of 
Financial Empowennent and several local banks or credit unions. 2 They were savings accounts 
that were overdraft-protected and did not have fees or minim um balance requirements. On the 
18-month parent survey, 55 percent of parents reported opening an Opportunity NYC account. 
Participants who could not or were not willing to use a bank account to receive reward pay- 
ments could have their payments loaded onto a stored value card. Seedco chose the Chase 
Payment Card. The cards could be used at ATMs for cash withdrawals and for point-of-sale 
purchases as a debit or credit card. Chase provided access to activity and balance information 
online or through monthly paper statements; however, most transactions and services that were 
associated with the card carried fees. According to Seedco data, only 1 1 percent of adults opted 
for the stored value card. 

In each family, a parent (or the legal guardian) — usually a single mother — received 
the payments for all health and workforce rewards earned by the family and any education 
rewards earned by elementary or middle school children. The parent also received half the value 
of some education rewards earned by high school students (for attendance, credit accumulation, 
and graduation) and the full value of the reward for attending high school students’ parent- 
teacher conferences. The program designers’ goal here was to reward parents for becoming 
more fully engaged with their children’s school performance. High school students, for whom 
financial incentives were expected to have more meaning than such incentives would have for 
younger children, and who controlled more of their own consumption behavior, had their own 
bank accounts or stored value cards into which they received payments for completing educa- 
tion activities. In addition to receiving half the value of some rewards, they received the full 
value of the rewards for having a library card, passing the Regents exams, and taking the 
Preliminary Scholastic Aptitude Test (PSAT). Because most high school students were minors, 
they were unable to open bank accounts without a parental cosigner. Some parents opened 
custodial accounts for their children. The result in both cases was that youth often needed their 
parent’s permission to access their funds, unless their parent chose to give them unfettered use 
of the account’s ATM card. 

The payment processing system was described in detail in the 2010 Family Rewards 
report. 3 It involved approximately 20 staff who used manual and automated processes to check 
and recheck each participant’s qualification for reward payments. Seedco staff rejected coupons 
when they were missing required signatures or documentation, or when the participant submit- 


2 The banks that offered Opportunity NYC accounts were Bethex Federal Credit Union, Brooklyn Cooper- 
ative Federal Credit Union, Carver Federal Savings Bank, Lower East Side Peoples Federal Credit Union, 
M&T Bank, North Fork Bank, and Union Settlement Federal Credit Union. 

3 See Chapter 3 and Appendix B in Riccio et al. (2010). 
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ting the coupon was not eligible for the reward. Participants were able to resubmit incomplete or 
incorrectly completed coupons for later payment using a coupon resubmission form. 

Every other month, families received an Earnings Statement with their payment, which 
summarized how much the entire family had earned in a particular payment period, and for 
which activities. One Earnings Statement was mailed to the family, addressed to the parent. As 
a result, many high school students relied on their parents for updates about their program 
perfonnance and bank account activity. In families where both the parent and student were 
engaged in the program, they became accustomed to the schedule of Family Rewards payments 
and had regular conversations, timed around the fifteenth of every other month or the arrival of 
the Earnings Statement, about what the student had achieved and failed to achieve. 

Seedco oversaw the entire payment processing component of the program, including 
staffing a telephone helpline and distributing the Earnings Statements. Responsibility for 
providing families with in-person support concerning coupon submission, setting up bank 
accounts, or payment problems was handled by two Family Rewards staff members located at 
each of the six participating NPOs. These staff also provided referrals to resources in the 
community that could help families meet the conditions for earning rewards (for example, for 
tutoring or employment assistance). They hosted biannual social events to bring together larger 
groups of families and to build a sense of community and identification with the program. In its 
intermediary role, Seedco provided technical assistance to the NPOs about ways to market the 
rewards to families within the limits of the “no case management” model. Seedco developed 
some marketing material centrally, such as the resources available on the Family Rewards Web 
site, “robocalls,” and newsletters. Seedco also supported the NPOs in developing flyers and 
letters that were mailed regularly to families, the scripts for telephone outreach campaigns, and 
small group workshops. 4 

Early Experiences Operating Family Rewards 

In the first year, Seedco and the NPOs succeeded in launching a professional operation 
with adequate capacity to meet the demands of serving almost 2,400 families, but they encoun- 
tered some challenges. Seedco established the payment processing system early on, but many 
other elements of the customer support and marketing strategy were delayed because the 
recruitment period overlapped with the start of program activities or because they were still in 
development. The 2010 Family Rewards report describes the uneven delivery of program 


4 In addition, MDRC played a technical assistance role, providing guidance to and general oversight of 
Seedco and the NPOs, in particular about how to adhere to and strengthen the program model. The Center for 
Economic Opportunity (CEO), MDRC, and Seedco worked closely together to make policy decisions and 
review program progress throughout the demonstration. 
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orientations across the NPOs, which may have influenced some participants’ initial understand- 
ing of the program. While the payment processing infrastructure was quickly put into place, the 
policies regarding acceptable forms of documentation for verifying certain rewards continued to 
evolve as Seedco and the program designers developed rules to cover unanticipated situations 
and tried to reduce administrative barriers to earning rewards. This was a particular issue in 
relation to the reward for full-time work because of the variety of pay stubs that people received 
and the prevalence of self-employment in this population. In sum, the first program year was the 
de facto “start-up” period because Family Rewards did not have a pre-research pilot phase, 
which affected some aspects of program delivery. 

Nevertheless, Seedco and the NPOs committed to a process of continuous program im- 
provement and were operating a mature and multidimensional program in the second year. 
Payment processing policies and procedures stabilized and became more routine and efficient. 
The payment processing team was thus able to troubleshoot some issues with coupon submis- 
sions without always rejecting the coupon and sending it back to the family. The program’s 
paperwork demands and deadlines became more familiar to the staff, who were then better able 
to advise families about ways to incorporate them into daily life. In addition, Seedco, MDRC, 
and CEO worked together to clarify the distinctions between case management and customer 
service for staff. This clarification had a liberating effect on many staff members and in fact 
expanded the breadth and variety of outreach. In the first year, staff were uncertain about 
whether they could give any personalized infonnation to participants for fear of crossing the 
line into case management, which was not part of the program design. Seedco regularly directed 
NPO staff to call all families to market the rewards without informing the staff whether a 
particular family had coupon rejections with which they needed assistance, or particular rewards 
they had failed to earn. By the second year, it was understood that Seedco could use its payment 
data to send out filtered lists of participants to NPOs so that they could engage in more targeted 
letter and telephone campaigns. Staff began contacting families with particular earnings 
histories and encouraging them to follow through on certain activities, like calling the helpline 
or attending a workshop at the NPO. Figure 2.1, which shows a flyer (with a fictional family 
name) that was mailed to participants, is an example of this kind of targeted outreach. Seedco 
mailed the flyer only to families who had coupon rejections in Year 2. The flyer instructs 
families to call the helpline to investigate and rectify the problem rather than go into the NPO 
for a one-on-one meeting (although many parents did that anyway). This example illustrates one 
of the ways in which a customer service-oriented approach differs from traditional case 
management. 

The NPOs held numerous workshops each month on topics that were meant to give 
families the resources to earn rewards. NPOs in the same borough also started to collaborate 
with each other by hosting joint health fairs in the spring, where participants could get health- 
related infonnation in one place. 
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The Opportunity NYC Demonstration: Family Rewards 
Figure 2.1 

Family Rewards Targeted Outreach Flyer 
Urging Families to Resubmit Rejected Coupons 




This is your last date 
to resubmit Year 2 Coupons * — ■> 

that were rejected 
and earn BIG REWARDS!!!! 


(? Call the Helpline Now 
and find out howto resubmit 



Ymi can download the Resubmission Coupon @ www.opportunitynyc.org/forms 


OPPORTUNITY © 


* Year 2 Coupons were for activities completed between 09/01/08 to 08/31/09 


Seedco’s engagement of the DCF Advertising firm to produce a Family Rewards 
Web site, newsletters, and other marketing materials was another key program develop- 
ment. These products were of professional quality (see Figure 2.2) and supplemented the 
many flyers created by the NPO staff each month. 

When program group families were first oriented to the program, they were told that the 
program would last two or three years, depending on funding. As a result, NPO staff had to 
inform families that the program had been extended when the decision was made to offer a third 
year in the summer of 2009. They usually spoke to parents, who, in turn, were expected to 
communicate that the program had been extended to their children. Staff informed parents 
through letters, the Family Rewards Web site, and messaging during a social event that was held 
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IS YOUR CHILD READY FOR THE 
BIGGEST TESTS OF THE YEAR? 

In January, all New York kids in grades 3 to 8 will take the 
English Language Arts (ELA) test. And in March, all kids 
will take the important Math test. The tests help decide if 
your child is ready to pass to the next grade. 

Helping your child prepare now pays big. Families can 
automatically receive up to $700 when a child passes the 
tests or improves scores. 

There’s a lot you and your child can do together to get 
ready. Now is the time to talk to your child’s teachers and 
start preparing. Your child’s future depends on you! 

! TCCT i-natcc- Grades 3-5: January 12th - 16th ! 
i IC3I UMI C5. Grades 6-8: January 20th - 23rd j 

VISIT OUR NEW WEB SITE OR CALL VOUR COMMUNITY 
ORGANIZATION FOR TIPS ON THE ELA AND MATH TESTS. 

WWW.OPPORTUNITYNYC.ORG 
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at the end of the second year. According to Seedco management reports, traffic on the Web site 
increased significantly (41 percent over the previous quarter) during this period, suggesting that 
families may have looked to the Web site for this notification. 

Program Implementation During the Final Year 

The third year of the program was a continuation of the program staffs confident im- 
plementation of the program model, but it also marked a time of transition because the rewarded 
activities changed and the entire intervention was coming to an end. 

Seedco and the NPOs continued to engage in creative and multifaceted fomis of out- 
reach to families. These activities included targeting advice and referrals to families who had 
coupon rejections or had not earned particular rewards, making monthly automated calls to 
remind families about upcoming activities, sending newsletters to families, and attaching 
informational inserts to the bimonthly Earnings Statements. The program staff occasionally sent 
e-mails to families that provided program infonnation or resources, such as infonnation about 
local college fairs for college-bound high school students. The e-mail distribution list consisted 
of 700 to 800 participants. 

Seedco had planned to install a module on each staff person’s workstation that would 
have allowed the staff to access current participant payment infonnation. This was considered 
an important tool to enhance the efficacy of the NPO staffs customer service role because they 
were often confronted with customers who were confused about why they had not earned a 
reward, but could not respond directly to their questions because payment decisions were made 
at Seedco. In these cases, the staff had to call the helpline to get more infomiation — just like 
the participants. Seedco worked with programmers for more than a year to create the module, 
but it was never successfully implemented. As a work-around, the staff was given a dedicated 
line to access helpline staff. 

In the third year, Seedco and the NPOs also added some new strategies to engage high 
school students. A large group of students who started Family Rewards in the seventh grade and 
were promoted on time every year entered high school at the start of Year 3. This cohort 
amounted to 72 1 additional high school students who needed their own bank accounts and to 
leam about the new educational rewards for which they would be eligible. The program staff 
hosted a “high school festival” at the Borough of Manhattan Community College in the fall of 
2009 to bring together high school students from all boroughs and to provide information about 
the educational rewards and tips for earning them. This event was successful, attracting 97 high 
school students and more than 80 parents and other siblings. 

Staff also called and sent flyers to the new high school students. An example of a 
flyer is provided in Figure 2.3. It incorporates the Web address for a special section of the 
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The Opportunity NYC Demonstration: Family Rewards 
Figure 2.3 

Family Rewards High School Student Outreach Flyer 



OPPORTUNITY © 
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To Do List: 

iSSSSSSSSS!^ 


Urban Health «»■ 


/ Attend activities r 11 ul 

Don't Forget to 

Friend us on MySpace: 

learn more about the program and get 
timely updates. Send a friend request, with 
your full name and family id #, to: 
www.myspace.com/ onyc_teen 
If you don't know your family ID, ask your 
parent or call the Helpline at (21 2) 994-452S 


Call Urban Health Plan at (347) 387-3747 
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Family Rewards Web site where students could find educational resources, a reminder about 
the upcoming Regents exams, the message that it was the last year of the program in order to 
encourage students to participate before time ran out, and an invitation to “friend” the pro- 
gram on MySpace. 

The drive to use social media to engage youth was a major initiative in the third year, 
coming out of the recognition that program staff had to find more direct ways of marketing to 
young people because parents were not always reliable or effective messengers, and teens did 
not often show up to workshops or social events. 5 NPO staff invited students to “friend” the 
program whenever the occasion arose, with one NPO even hosting a “MySpace Party.” Despite 
these efforts, the MySpace page was never a dynamic or much-frequented site, and did not get 
many users. The small number of students who commented on their lack of interest in the site 
during in-depth interviews with MDRC researchers or conversations with NPO staff said that 
they did not use MySpace because they had moved on to Facebook, Twitter, or other chatting 
applications; they did not think the site was informative or interesting; or they simply did not 
want their affiliation with the program to be part of their public profile. 

Two Significant Program Changes in Year Three 

There were two significant changes in the third year of the program that had implica- 
tions for reward earnings: (1) the roll-out of an exit campaign, which started with a campaign to 
get participants to recognize that their chance to earn rewards was ending, and (2) the elimina- 
tion of several rewards. 

The campaign to prepare families for exit 

Because it was the final year of the program, Seedco and the NPOs aimed to send the 
message that families should make the most of this last opportunity to earn money from the 
program, and to prepare participants for a return to life without the income boost provided by 
Family Rewards. 6 These messages permeated most written materials that Seedco and the NPOs 
distributed in the last year. Figure 2.4 provides an example of this type of messaging in a 
newsletter. The title reads, “The Countdown Begins!” (Seedco did eventually put an image of a 
“countdown clock” on the Family Rewards Web site that indicated the number of days remain- 
ing in the program.) It goes on to say, “With only a few months left, now is the time to focus on 
earning all the Rewards you can and start planning for life after the program.” Parents were 


5 Greenberg, Dechausay, and Fraker (2011). 

families could earn rewards for activities completed before September 1, 2010, and had until May 15, 
2011, to resubmit rejected Year 2 coupons or turn in employment documentation if they were self-employed. 
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The Opportunity NYC Demonstration: Family Rewards 


Figure 2.4 

Family Rewards Flyer Preparing Families for the End of the Program 


OPPORTUNIT |yC 

family rewards 


newsletter in this issue: Tips for Teens and Parents 


The Countdown Begins! 

With only a few months left, now is the time to focus on earning 
all the Rewards you can and start planning for life after the program. 
Ask yourself these questions to get started. 


After 3 
great years, 
f ONYC will end on 
August 31, 2010. 

The deadline for 
^ Year 3 coupons is 
September 15, 
2010. 


Has my family had their annual 
doctor and dentist checkups. 

sessso&i 

ssassawesu 

earn big Rewards for doing it, too. 


Are my children prepared for 
the ELA, Math or Regents tests? 

All children in grades 3-12 have some 
important standardized tests coming 
up. How well your children do on these 
tests helps their schools make decisions 
about their education. Success can earn 
your family big Rewards, too. Talk to 
your community organization about 
finding practice tests or go to www. 
OpportunityNYC.org for more resources. 


How am I using my Fami| 
Reward payments now? 

Thinking about how you're 

Reward payment* V 6 ng your 

for when they stop cLy 0 help> y ° U pre P ar e 

your community ZanZT Y ° U ta,k *o 

financial planning resourceTand ^ 




invited to ask themselves whether they had completed and submitted coupons for their preven- 
tive health care visits, whether their children were prepared for the State exams, and how they 
were spending their current Family Rewards payments. Other flyers created that year encour- 
aged parents to engage in more intentional financial planning and to focus on finding employ- 
ment or training opportunities, since they would not be able to maintain the quality of life they 
had experienced during the program unless they could work more hours or advance at work. 

Families generally reacted to the end of the program with acceptance. They expressed a 
mix of sadness, gratitude for having had the experience, and, in some cases, a desire to save 
more money in the final year to make the rewards last longer. One mother explained the end of 
the program to her two daughters, drawing on these themes: 

1 told them, “Okay, girls, we have to give thanks to God because we were so 
privileged with this program that has helped us so much. So, with what’s left and 
what’s still on the way, we have to figure out how we’re going to invest it and 
save it and stretch it as far as possible, because the program is going to end.” 

“Oh, mom, that’s not good. Why?” 

1 said, “Well, you know how the economy is and everything. We have to 
give thanks to God because we got a lot out of this. We were very blessed. So, 
we have to know how we’re going to invest what’s on the way.” 

Another mother had a simpler message that focused on being resilient rather than mak- 
ing plans for the program end: “I think I just sat them down, or in passing, I was just, like, you 
know, the program is about to end and things are gonna change. This is part of life. In life, you 
have to adjust to changes.” 

While some parents tried to shift their families’ spending habits away from spending on 
“extras” to spending only on necessities, the parents who were interviewed described few 
strategies to earn more money from the program in its final year. 

The centerpiece of the campaign to prepare families for the end of the financial rewards 
was a special section of the Family Rewards Web site that contained resources and worksheets 
designed to help families set goals for the future and access appropriate community resources. 
Seedco began developing the content of the site in early 2010 and it went live in mid-July. It 
listed important final deadlines regarding coupon submission and a resource guide that included 
contact information for an array of programs that offer services related to education, health, 
workforce development, and financial education throughout the five boroughs. It also included 
practical suggestions for both adults and students about money management. The information 
on the Web site was available in both Spanish and English, although some of the links to 
external services were only available in English. During the third quarter of 2010 (July through 
September), the exit section of the Web site received only 441 page views, about 4 percent of all 


37 



page views. 7 None of the parents who were interviewed at the close of the program described 
using the exit resources. 

There are a number of reasons the exit materials may not have been used more. The 
Web site was launched relatively late in the program year. Seedco noted that in the months 
preceding the launch of the exit page (April to June), the Web site received 4,296 visits, most of 
which were to the education and training page for parents (35 percent) and an advertisement for 
the National College Fair (41 percent), suggesting that participants may have begun to search 
for resources to cope with the post-program period earlier. At the end of the program, partici- 
pants who had used the Web site (52 percent of those asked on the 42 -month survey) may have 
believed that they knew what kinds of infomiation they could get on the site because its struc- 
ture had not changed much over the course of the program, so they may not have expected to 
find much new content. 8 For those who did visit the exit page, it contained the equivalent of 
many pages of text, which placed heavy demands on literacy and may have limited accessibil- 
ity. Last, the central theme of the exit strategy was that families should be encouraged to 
maintain the new habits they had formed during the program years and find ways to replace the 
extra income provided by the program by seeking assistance from other community agencies. 
For families who greeted the end of the program with resignation or indifference, or who 
required more intensive assistance to overcome their barriers to accessing appropriate supports, 
this message may not have resonated. 

The fact that the exit materials on the Web site do not appear to have been used much 
does not mean that many families were not thinking about and preparing for the end of the 
program. Parents were asked on the 42 -month survey whether they had ever attended a work- 
shop or received referrals from the NPO in Year 3. Their answers demonstrate increased use of 
those services compared with the responses on the 18-month survey. At 18 months, 43 percent 
of parents had ever attended at least one workshop, and about one-fourth or less had received 
referrals for workforce development, educational services, or health care services. On the later 
survey, about half of parents had attended a workshop, and 57 percent had received a referral in 
the final program year. Of the parents who received referrals, most requested referrals to 
advance their own education and training or to get supplemental educational resources for their 
children, as shown in Table 2.1. It is impossible to know how many of those referrals were 
explicitly prompted by parents’ desire to prepare themselves and their children for the end of the 
program, but it is likely that some of those referrals worked to support post-program readiness. 


7 A11 data about Web site visits are from Seedco management reports. 

8 Not all the content on the exit page was new. One of the links on the exit site brought users to the Educa- 
tion and Training resources, which could be accessed from the “Work” tab on the home page. 
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Table 2.1 


Participants’ Experience of Family Rewards Program Services 


Outcome 

Program 

Group 

Number of times called NPO staff for information about 
Family Rewards, compared with the first 2 years (%) 

More often 

20.8 

Less often 

40.9 

About the same 

35.4 

Never 

2.8 

Payment processing decisions were (%) 

Consistent over time 

84.6 

Prompt and efficient 

73.6 

Fair 

90.7 

All of the above 

64.1 

NPO staff made any referral a (%) 

57.0 

Health care provider or dentist 

27.5 

Education or training program 

36.3 

Program to help find a job 

31.5 

Tutors, after-school activities, or programs to help children 

with school 

37.8 

Information about colleges for your child 

31.8 

Attended any Family Rewards workshop (%) 

50.3 

Visited the Family Rewards Web site for information (%) 

51.5 

Sample size 

717 


SOURCE: MDRC calculations using data from the Family Rewards 42-month survey. 

NOTES: This table excludes control group members because it pertains only to the 
Family Rewards program. A randomly selected subsample of program group members 
was asked these questions. 

Sample sizes may vary because of missing values. 

NPO = Neighborhood Partner Organization. 

Percentages may sum to more than 100 percent because sample members could list 
more than one response. 


The elimination of several rewards 

The second major development in the last year of the program was that several rewards 
that had been available in the first two years were discontinued. Those rewards were for school 
attendance (for students in first through eighth grades); discussing results of the annual English 
language arts (ELA) and math tests with teachers (for students in third through eighth grades); 
having a library card (for students in all grades); and maintaining public or private health 
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insurance and completing a doctor-recommended follow-up visit within a specified time frame 
(for all family members). In addition, the education and training reward was modified so that 
parents would not have to work to qualify for it, and the reward that parents received for 
reviewing the results of low-stakes interim tests (for students in the first through eighth grades) 
had been discontinued in Year 2. 

Removing the part-time work requirement from the training activity made it easier for 
parents to qualify for the training reward. In the other cases, the elimination of the rewards 
removed an opportunity for families to earn money through the program. Program designers 
eliminated those rewards to focus the incentive schedule on activities for which there was 
sufficient opportunity to make an impact through the incentives (reflected in the disparity 
between the control and program group outcomes), and to remove rewards that were too time- 
consuming to verify or too difficult for families to complete. They also hoped that a shorter 
list of rewards would help families internalize the activities and sharpen their focus on 
particular behavioral outcomes. Interestingly, the change in the incentives schedule created 
the conditions for an early assessment of how families would respond to the removal of the 
financial incentives. 

There was some risk that changing the incentives schedule at this stage would be con- 
fusing for families or even spark resentment toward the program. To address these concerns, 
Seedco and the NPOs explained the changes openly when they distributed coupon books at the 
start of the third year, and they created a special brochure explaining the changes. What was 
missing from this outreach to families was an explicit discussion of how the elimination of these 
rewards was likely to affect families’ ability to earn the same amount of reward money as they 
had in the past — in short, that families would have to approach the program in a different way 
to maintain their same level of earning. This could have been part of the messaging around the 
exit campaign as well, but it was not. 


Families’ Understanding and Use of the Program 

The Family Rewards study has tracked how participants’ understanding of the incentives has 
evolved over time. This is important because the program’s theory of change requires that 
participants (1) know what activities and performance levels are being rewarded, (2) are motivat- 
ed by the offer, (3) have the intention and resources to accomplish the activities, and (4) are able 
to complete the verification procedures (that is, fill out and mail coupons when needed). 9 


9 This is not necessarily the case when it comes to young children. Parents can play a role motivating and 
supporting young children’s educational achievement even when the children don’t know about the financial 

(continued) 
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The extensive marketing of the rewards and the reduction of the number of rewarded 
activities in the third year were efforts to make it easier for families to familiarize themselves 
with and remember the rewards. The 18-month parent survey found that parents had a good but 
overly broad understanding of the rewards. They tended to believe that more activities were 
rewarded than was the case and had poorer recall of the specific levels of perfomiance required 
for automatically verified rewards, suggesting that they relied heavily on the coupon book for 
program information. Ninety-three percent of parents said they knew what they needed to do to 
submit coupons for activities they had completed, although a lower proportion — 63 percent — 
said that they had no trouble keeping track of the coupons. 

Seedco management data on parents’ use of the helpline and on parents’ responses to 
the 42-month survey provide some insight into how parents’ understanding changed over 
time. Helpline call volume generally decreased over the course of the program. In Year 3, 
when participants called, it was usually to check to make sure their coupons had been re- 
ceived or to inquire about when a payment would be posted to their accounts. Parents report- 
ed on the survey that they sought help from NPO staff in the third year of the program the 
same amount as or less often than they had in the first two years, as shown in Table 2.1. The 
declining reliance on customer service could be seen as a sign of growing independence and 
comfort with the program mechanics. On the other hand, it could simply indicate lower levels 
of engagement as the program drew to a close. The balance of evidence suggests, however, 
that most families understood and were able to handle the paperwork requirements of the 
program by the time it ended. 

When asked for their assessment of the payment processing system, most families rated 
it positively. Ninety-one percent of parents described payment decisions as fair, 85 percent said 
they were consistent over time, and 74 percent thought payments were received in a prompt and 
efficient manner. The fact that one-fourth of parents did not consider the payment process to be 
prompt and efficient is a concern, however, because the more removed in time the payments are 
from the behavior, the lower will be the expected reinforcement effects of the cash reward. 10 

A key question for the implementation research is whether families had particular diffi- 
culty submitting valid coupons for any of the activities. One of the reasons some activities were 
eliminated during the program and the rules for submitting others were changed was to remove 
activities that were not practicable and to reduce administrative barriers to successfully claiming 
rewards. Seedco program data on coupons that were rejected but never resolved was used to 
probe this issue. It shows that 1,915 families (about 80 percent of all families in the program) 


rewards. Indeed, the implementation study found that young children sometimes were not told that their family 
was part of Family Rewards. 

10 Skinner (1974). 
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had outstanding rejections at the close of the program. Sixty-one percent of rejected coupons 
were in the health domain, 24 percent were in the workforce domain, and 1 5 percent were in the 
education domain. Table 2.2 lists the activities that had the highest number of unresolved 
coupon rejections. The coupons for full-time employment for adults and for maintaining private 
health insurance for children and adults top the list. All the activities that are listed in Table 2.2 
were eliminated at the start of Year 3, with the exception of the coupons for full-time work and 
for dental care. 11 

There is a temporal pattern to the volume of coupon rejections, which is illustrated in 
Figure 2.5. Most coupons that were rejected and never subsequently approved were received at 
the transition points between program years. Specifically, 9 percent of unresolved coupon 
rejections occurred in the first payment period, 10 percent in the sixth payment period, and 1 1 
percent in the twelfth payment period. At the start of the program, families were likely to make 

Table 2.2 


Activities for Which Coupons Were Rejected 
Most Often and Never Resolved 


Activity 

Coupon Rejections, 
Number 

Coupon Rejections, 
as Percentage of Total 

Sustained full-time employment 

2,803 

19.4 

Maintaining private health insurance for children 

2,212 

15.3 

Maintaining private health insurance for adults 

2,074 

14.4 

Preventive dental care for children older than age 6 

1,485 

10.3 

Completing a doctor-recommended follow-up 
visit for children 

886 

6.1 

Preventive dental care for adults 

723 

5.0 

Education and training while employed 

701 

4.9 


SOURCE: MDRC calculations using Seedco's Family Rewards program data. 

NOTE: These calculations show the number and percentage of coupons that were never approved 
for health and work-related activities that participants claimed to have completed between 
September 2007 and August 2010. 


"Coupons for education and training while employed were also continued. 


42 







The Opportunity NYC Demonstration: Family Rewards 
Figure 2.5 

Number of Coupon Rejections for the Full Sample, by Payment Period 



Payment period 

SOURCE: MDRC calculations using Seedco's Family Rewards program data. 

NOTE: The first program year covers September 2007 through August 2008; the second program year covers September 2008 through August 
2009; and the third program year covers September 2009 through August 2010. 



mistakes as they were getting to know the program rules. At the end of each program year, NPO 
staff launched massive campaigns to get families to submit all their eligible coupons and fix any 
outstanding errors. As a result of those campaigns, families submitted large numbers of coupons 
in the last payment period of the year, but it is clear from the volume of rejections that many of 
those coupons were not successfully redeemed. 

With each passing payment period, the volume of permanently rejected coupons de- 
creased, suggesting that parents became more competent at submitting valid coupons. A very 
small amount of the decrease may also be a result of attrition, as parents who experienced 
coupon rejections or other frustrations with the program stopped participating. 12 

Overall, these findings highlight the trade-offs that are involved in using coupon-based 
incentives. In order to fill out the coupons, participants had to leam program rules and navigate 
systems, like their Human Resources departments. The program staff invested significant 
resources explaining the verification rules and addressing rejected coupons. At the same time, 
the coupons provided a tangible reminder of the incentives that many participants appreciated 
and were manageable for most. These advantages and disadvantages need to be taken into 
consideration when designing the verification process for an incentives-based intervention. 


Money Earned from Family Rewards 

Family Rewards transferred a significant amount of additional income to poor families in New 
York City in exchange for their performance of certain activities. The amount of the average 
cash transfer to a household is on the order of the Earned Income Tax Credit (EITC). 13 Cumula- 
tive reward earnings over three years averaged $8,707, with the majority of families (71 
percent) earning a total of more than $5,000 from the program (Table 2.3). There are important 
differences to note between the EITC and Family Rewards, however. Unlike the EITC, which is 
paid out in one lump sum once a year, the Family Rewards payments were made every two 
months in sums that varied based on which activities families had completed. Earnings from 
Family Rewards were combined with other sources of revenue in participants’ bank accounts or 
on stored value cards, and they were split between parents and high school students. Paying out 
rewards periodically likely increased the salience of the program for families, but the variable 


^Analyses show, for example, that the bottom 10 percent of earners had most rejections in the beginning 
of the first year of the program, and subsequently very few rejections. Their overall earnings, in conjunction 
with this finding, suggest that these families stopped submitting coupons. 

l3 The EITC is a tax credit for low-income taxpayers with qualifying children. The Internal Revenue Ser- 
vice (IRS) reported that the average amount refunded for the 2010 tax year was $2,240 per family. See 
www.eitc.irs.gov/central/eitcstats. 
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Table 2.3 


Summary of Rewards Earned by Families 


Outcome 

Year 1 

Year 2 

Year 3 

Years 1, 2, and 3 
Combined 

Family earned at least one reward (%) 

99.5 

98.0 

89.1 

99.6 

Only automatically verified rewards 

16.5 

18.2 

12.0 

9.8 

Any automatically verified rewards 

98.7 

96.7 

73.7 

99.3 

Only coupon book rewards 

0.8 

1.3 

15.4 

0.3 

Any coupon book rewards 

83.1 

79.8 

77.2 

89.9 

Automatically verified and coupon book rewards 

82.2 

78.5 

61.8 

89.6 

Average number of automatically verified 

rewards earned 

14.0 

13.0 

3.4 

30.4 

Distribution of number of automatically verified 
rewards earned (%) 

0 

1.4 

3.3 

26.3 

0.7 

1-10 

32.7 

34.8 

69.5 

7.7 

11-20 

47.2 

47.5 

4.3 

20.2 

2 1 or more 

18.7 

14.4 

0.0 

71.4 

Average number of coupon book rewards earned 

13.6 

12.0 

7.9 

33.4 

Distribution of number of coupon book 
rewards earned (%) 

0 

17.0 

20.2 

22.8 

10.1 

1-10 

27.2 

30.2 

43.4 

12.5 

11-20 

31.4 

28.1 

29.7 

14.1 

2 1 or more 

24.5 

21.5 

4.0 

63.3 

Family earned at least one reward (%) 

Education reward 

96.4 

91.5 

82.5 

97.9 

Health reward 

95.2 

94.2 

72.7 

98.0 

Workforce reward 

42.2 

42.1 

41.9 

53.2 

Among families who earned a reward in a specified 

period, average reward amount earned ($) 

3,153 

3,196 

2,700 

8,707 

Average reward amount earned, by domain 11 ($) 

Education 

1,450 

1,477 

1,394 

3,983 

Health 

1,224 

1,247 

879 

3,039 

Workforce 

1,359 

1,438 

1,475 

3,376 

Average number of activity periods 

during which rewards were earned 

5.6 

5.6 

4.2 

14.8 


(continued) 
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Table 2.3 (continued) 


Outcome 

Year 1 

Year 2 

Year 3 

Years 1, 2, and 3 
Combined 

Distribution of average reward amount earned b (%) 

$1 - $99 

0.6 

0.6 

0.3 

0.2 

$100 -$499 

4.7 

5.2 

7.1 

1.1 

$500 - $999 

7.9 

7.2 

9.7 

2.1 

$1,000 -$2,999 

39.6 

37.9 

43.7 

11.2 

$3,000 - $4,999 

29.7 

31.3 

28.2 

14.5 

$5,000 - $6,999 

12.1 

13.2 

9.1 

13.6 

$7,000 or more 

5.4 

4.6 

1.8 

57.4 

Family picked up coupon book (%) 

89.8 

86.9 

87.0 

96.0 

Sample size 




2,377 


SOURCE: MDRC calculations using Seedco's Family Rewards program data. 

NOTES: The first program year covers September 2007 through August 2008; the second 
program year covers September 2008 through August 2009; and the third program year covers 
September 2009 through August 2010. 

a Reward amounts are calculated among families who earned rewards for each domain during 
the specified period. 

b The maximum amount earned in Year 1 was $13,235; in Year 2, it was $12,150; and in Year 
3, it was $10,425. 


amounts of the payments affected families’ ability to budget and account for the Family 
Rewards earnings, a topic that is discussed below. 

Table 2.3 summarizes the rewards that families earned each program year and overall. 
In general, families showed a high level of participation in the program to the end. In Year 3, 89 
percent of families earned at least one reward, and average annual earnings totaled $2,700. 
Seventy-two percent of families earned between $ 1 ,000 and $5,000 that year. 

At the same time, according to a variety of indicators, reward receipt generally declined 
between the second and third program years, with the exception of workforce rewards. Families 
earned rewards in fewer payment periods in Year 3 — an average of 4.2 out of 6 payment 
periods, compared with 5.6 payment periods in Year 2. They earned fewer automatically 
verified rewards — an average of 3.4 in Year 3 compared with 13 in Year 2. Twenty-six 
percent of families did not earn any automatically verified rewards in Year 3. The third year saw 
a decline in the average number of coupon-based rewards earned as well, although not as steep. 
About 88 percent of families picked up their coupon books each year, and 79 percent of families 
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picked up their coupon books in all three years (analysis not shown). 14 The fact that families 
continued to get their coupon books and submit coupons is a sign that they continued to be 
invested in the program, since managing the administrative demands of coupon submission 
required attention and planning. Still, because the decline in automatically verified rewards was 
not met with an increase in the completion of coupon-based rewards in Year 3, families did not 
sustain the total dollar value of the transfers they had earned in the first two years of the pro- 
gram. On average, families earned $496 (or 16 percent) less in the last year of the program than 
they did in Year 2. 

Rewards Earned in the Education, Health, and Workforce Domains 

Families earned a total of $20.6 million from Family Rewards over three years. 15 Of 
this total, 45 percent was paid for education rewards, 34 percent for health rewards, and 21 
percent for workforce rewards, as Figure 2.6 illustrates. Almost every family earned at least one 
education and health reward, as shown in Table 2.3. About half of families earned at least one 
workforce reward. The level of participation in the workforce domain did not change over the 
course of the three years of the program; however, the proportion of families who earned at least 
one education or one health reward in Year 3 compared with Year 2 declined by 9 percentage 
points and 22 percentage points, respectively. In the final program year, 83 percent of families 
earned at least one education reward, 73 percent earned at least one health reward, and 42 
percent earned at least one workforce reward. 

The education rewards for high school students had the highest monetary value of any 
category of rewards. High school students directly earned about $3 million in total for educa- 
tional achievements, an amount that was deposited into bank accounts that students’ cosigned 
with their parents or onto stored value cards in the student’s name. In an attempt to estimate the 
relative contribution of high school students’ education rewards to family earnings, Table 2.4 
shows the proportion of average annual family earnings that was made up by ninth-graders’ 
educational rewards. 16 This table contains only the subgroup of program group families who 
had at least one ninth-grader at the start of the program and shows only the contribution of that 


14 Coupon book distribution continued for the entire program year. In Year 3, 77 percent of coupon books 
were distributed by December 31, 2009. This was the most efficient pace of distribution of any year. While it 
was possible to get individual coupons from NPO staff and the Web site, families who did not pick up their 
coupon books were likely to miss out on the opportunity to earn coupon-based rewards. 

15 Most of these payments were successfully made to families’ accounts. This did not always occur every 
two months, as described earlier. Sometimes there were delays retrieving administrative data from the 
Department of Education, which delayed those payments. Some participants did not receive all the money they 
had earned from the program because of problems with their bank accounts that were never resolved, or 
because they never submitted a bank account number or signed up for a stored value card. 

16 In the tables and throughout this report, “baseline” refers to the time of random assignment. 
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The Opportunity NYC Demonstration: Family Rewards 
Figure 2.6 

Total Amount of Reward Money Earned by Families in the Three 
Program Years Combined, by Domain 



SOURCE: MDRC calculations using Seedco's Family Rewards program data. 

NOTE: The first program year covers September 2007 through August 2008; the second 
program year covers September 2008 through August 2009; and the third program year covers 
September 2009 through August 2010. 

student’s rewards even if there are other high school students in the household. Additionally, the 
analysis does not break down how the education reward money was divided between the 
parents’ and the children’s accounts. 

Nonetheless, it is striking that in the first two years of the program, the starting ninth- 
graders’ rewards constituted one-third of overall household earnings, and in the third year, they 
were more than 40 percent. The 2010 report found that Family Rewards produced higher 
education outcomes for ninth-graders who had scored at proficiency level on their math or ELA 
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Table 2.4 


Proportion of Annual Reward Earnings per Family 
Made Up of Education Rewards Earned by Students in Grade 9 
at the Time of Random Assignment, by Math Proficiency 


Sample and Program Y ear 

Average Total Reward 
Earnings per Family ($) 

Education Reward 
Earnings (%) 

All families with students in grade 9 at baseline 

Year 1 

3,578 

31.1 

Year 2 

3,531 

30.3 

Year 3 

3,026 

42.1 

Years 1, 2, and 3 combined 

9,846 

34.4 

Sample size 


910 

Proficient on 8th grade math test a 

Year 1 

4,663 

40.5 

Year 2 

4,558 

38.6 

Year 3 

3,613 

47.5 

Years 1, 2, and 3 combined 

12,637 

42.5 

Sample size 


295 

Not proficient on 8th grade math test a 

Year 1 

3,086 

28.6 

Year 2 

3,058 

27.9 

Year 3 

2,757 

42.1 

Years 1, 2, and 3 combined 

8,569 

32.6 

Sample size 


531 


SOURCE: MDRC calculations using Seedco's Family Rewards program data. 

NOTES: Calculations are based on families who earned any rewards. 

Education reward earnings are presented as a percentage of average total reward earnings per 
family. 

a ln New York State, students who score at a level of 3 or higher on a 4-point scale are deemed 
"proficient." 


exams in eighth grade. 17 Looking specifically at families with a ninth-grader who was proficient 
in math, 18 the education earnings were about 43 percent of family earnings over the entire 
program, and peaked at almost half of family earnings in Year 3. For families with a ninth- 


17 Riccio et al. (2010). 

ls Table 2.4 shows findings for students whose proficiency is measured by their eighth grade math scores. 
Appendix Table B.l shows the same analysis as measured by eighth grade ELA scores. 
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grader who was not proficient in math, the education earnings still accounted for a meaningful 
proportion of household earnings, making up about 33 percent of family earnings over the entire 
program and 42 percent of earnings in the final program year. 

To put the amount of money transferred to high school students into context, it is im- 
portant to recognize that parents exercised varying degrees of control over how much access 
students had to their rewards. When parents were asked about this on the 42-month survey, their 
answers varied depending on the age of the child, a finding confinned by the in-depth inter- 
views described in MDRC’s qualitative report on Family Rewards. 19 The vast majority of high 
school students — 72 percent — had to ask their parents for pennission to spend money they 
had earned through the program. 20 Only 17 percent were able to spend the reward money as 
they chose, and 9 percent were not allowed to spend it at all. 21 


Explaining Patterns of Reward Receipt 

Why Families Earned Less Money in Year 3 Than in Earlier Years 

It was noted above that families earned, on average, $496 less in the last year of the 
program than in Year 2. Why did earnings decrease in the final year despite efforts by the 
program operators to get families to make the most of their last chance to earn rewards? The 
main reason is the elimination of several rewards. Figure 2.7 shows the proportion of Year 1 
and 2 average annual earnings that reflected the subsequently discontinued rewards. Public and 
private health insurance, which mostly reflected the automatically verified public health 
insurance reward, accounted for about one-fifth of annual earnings during that time. For 
families earning the least from the program, the health insurance rewards accounted for more 
than a third of their reward earnings in the first two years. 

By removing the reward for maintaining public health insurance, the designers removed 
a reward that provided consistent payments and turned out to be relatively easy to earn. As a 
consequence, the incentives schedule became more rigorous in the third year and required 


19 Greenberg, Dechausay, and Fraker (2011). 

20 A supplemental study of a sample of high school students and parents in the control and program groups 
explored spending and saving practices within families. See Morris, Aber, Wolf, and Berg (2012), pp. 37-41. 

21 Not all younger children even knew that they were earning money from the program or had access to it. 
Seventy-seven percent of elementary and middle school-age children knew about the money they earned from 
Family Rewards (though often not the specific amounts) and had access to it. Sixty percent of those parents 
required that their children get their permission before making purchases. Thirty-four percent allowed their 
children to spend as they pleased. (Most of the children in this last category are probably middle school 
students.) 
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The Opportunity NYC Demonstration: Family Rewards 
Figure 2.7 

Composition of Annual Earnings, Full Sample and Bottom 20 Percent of Earners 
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participants to take more deliberate steps to earn rewards and make up for the loss. It appears 
that families did not make those adjustments. This could be because the families were indiffer- 
ent to the loss of income, being accustomed to ups and downs in their financial stability, or 
because they did not fully grasp the impact of losing the health insurance reward and needed 
more support in order to cope with that change in a proactive way. 

One of the findings from the in-depth interviews is that parents and youth who were 
more organized were better able to both handle the verification procedures associated with the 
program and track their own perfonnance to increase the motivational power of the rewards. 
For example, one Brooklyn mother explained that she calculated how much she was expecting 
to earn every two months before she got the Earnings Statement, and followed up with the 
helpline or her children whenever there was a discrepancy. She set up payment plans with her 
creditors based on her expected reward earnings. She was one of the few participants who was 
able to accurately state how much her family had earned from the program over three years. 
Most interviewees were not able to estimate this amount, with most vastly underestimating the 
amount of the cash transfer. 

There are many reasons for this that relate to the concept of mental accounting, which is 
a broad concept that describes how people think about money, categorize it into different 
accounts or budgets, and use those categorizations to make decisions. 22 One of the findings 
within this area of research is that small dollar amounts are not often noticed, getting assigned to 
the “petty cash” category that is not mentally recorded. Although the annual transfer to families 
was sizable, it was disbursed in small payments of variable size, and many parents and youth 
reported that they did not recognize how much they were actually earning or their earning 
potential. 23 Parents who were not independently engaged in cumulative accounting of their 
reward earnings over time, like the mother described above, may not have even recognized that 
their level of earning had decreased in Year 3 until it was too late. The failure to do cumulative 
accounting of reward receipt may also have affected the likelihood that participants would place 
rewards into an incidental category of expenditures — such as for a monthly bill or article of 
clothing — rather than recognizing the potential for the money to be saved for a larger invest- 
ment, like a computer or debt repayment. 

Why Some Families Earned More Rewards Than Other Families 

Adding up reward earnings over three years reveals a considerable range in the amount 
of money families earned from the program. On average, families earned $8,700 overall, but the 
highest-earning family earned $32,820 in rewards while the lowest-earning family earned $0. A 


22 Thaler (1999). 

The Earnings Statement did not provide a tally of rewards earned to date. 
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large majority of families — approximately 80 percent — earned more than $3,375 over the life of 
the program, while the top 20 percent earned more than $13,000 (and an average of over $17,000) 
in reward money. It is impossible to fully explain the mechanisms that were responsible for the 
range in earnings outcomes because families varied widely in their composition, attributes, and 
internal dynamics, but certain baseline and program engagement characteristics are associated with 
being a low, moderate, or high earner in Family Rewards. 

The Association Between Baseline Characteristics and Reward Receipt 

Family size and composition were significantly correlated with earnings. The larger the 
family — especially the more high school students — the more opportunities it had to earn re- 
wards. Table 2.5 demonstrates this pattern. For each additional child up to three, families earned 
about $2,000 more on average. Families with no high school students earned about $2,800 less 
than families with at least one high school student. Having two adults enrolled in the program was 
another factor contributing to reward earnings (not shown in table), but few families had two adults 
(only 6 percent at the time of enrollment). 

The Opportunity NYC Demonstration: Family Rewards 

Table 2.5 


Variation in Average Amount of Reward Money Earned, 
by Family Size and Composition 


Average Rewards Earned and Family Composition* 

Years 1, 2, and 3 
Combined ($) 

Percentage of 
Total Sample 

Familes with 



1 child 

6,120 

22.8 

2 children 

8,121 

33.3 

3 children 

10,283 

26.7 

4 children or more 

10,891 

17.2 

No high school students 

7,311 

49.5 

At least 1 high school student 

10,078 

50.5 

Sample size 


2,368 


SOURCE: MDRC calculations using Seedco's Family Rewards program data. 


NOTES: The first program year covers September 2007 through August 2008; the second 
program year covers September 2008 through August 2009; and the third program year covers 
September 2009 through August 2010. 

"Sample size" refers to the number of families who earned rewards. 

Data for number of children were missing for 8 families at the time of random assignment. 
“Calculations are based on families who earned any rewards. 
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Table 2.6 illustrates the relationship between household size and the total amount of 
rewards earned. The top 20 percent of families, in terms of earning rewards, generally had more 
children than did other families. These larger families averaged four members (one adult and 
three children), with one of the three children being in high school. In comparison, families in 
the middle or bottom of the distribution had around three family members (two children) and 
were less likely to have a high school student. 

Although all families were eligible for the study because they qualified for free school 
lunch and lived in communities of concentrated poverty, the families who earned more rewards 
were less disadvantaged at the time of enrollment than families who earned fewer rewards, and 
they made more extensive use of the helpline and NPO staff. The top 20 percent of earners had, 
on average, higher levels of education, employment, earnings from employment, marriage, and 
self-reported mental and physical health than did other families at the time of enrollment, as 
shown in Table 2.6. They were less likely to be receiving government transfer benefits or slightly 
less likely to be living in very unstable housing (for example, with a friend or in a group shelter). 
The heads of household in these families were slightly younger than those who earned less. 

Earnings patterns were generally similar among black and Hispanic participants. Fami- 
lies who spoke Spanish at home were at no disadvantage when it came to earning rewards, 
likely due, at least in part, to the availability of bilingual NPO and helpline staff and translated 
program materials. Families with a foreign-born head of the household constituted 45 percent of 
the highest-earning group. A separate regression analysis also found that households with 
foreign-bom parents received higher reward amounts, even after controlling for other baseline 
characteristics, 24 suggesting that immigrant families adapted especially well to the intervention. 

All these attributes were strongly correlated with placement on the reward earning dis- 
tribution when the same analysis was done with reward receipt data from Years 1 and 2. Does 
this mean that reward earnings potential was predetermined by characteristics that existed 
before random assigmnent and remained fixed over three years? Additional analyses were done 
to look at earnings mobility between Years 1 and 3. About 41 percent of families were in the 
same quintile on the earnings distribution in Year 3 as they were in Year 1; 29 percent moved 
to a higher quintile; and 30 percent moved to a lower quintile. There were 477 families who 


24 The regression analysis controlled for household-level baseline characteristics, including ethnicity, pri- 
mary language in the household, citizenship, household composition, parents’ educational attainment, public 
assistance receipt, mental and physical health, connection to mainstream banking, and the focal child’s 
standardized test scores. 
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The Opportunity NYC Demonstration: Family Rewards 

Table 2.6 


Characteristics of Families Who Earned Rewards, 
by Range of Earnings During Program Years 1, 2, and 3 Combined 


Range of Earnings 

Characteristic 

Bottom 20%, 
$0 - $3,375 

Middle 60%, 
$3,376 -$13,125 

Top 20%, 
$13,126 - $32,820 

Characteristics of families at baseline 




Average number 




In household 

2.9 

3.4 

4.2 

Of children enrolled in Family Rewards 

2.0 

2.4 

3.0 

Of high school students in household 

0.5 

0.6 

1.0 

Primary language spoken at home is Spanish (%) 

18.9 

21.7 

25.5 

Family was receiving no government transfer 




benefits (%) 

15.0 

18.1 

38.1 

Sample size (total = 2,377) 

476 

1,427 

474 

Characteristics of parents at baseline 3 




Average age (years) 

41 

40 

39 

Race/ethnicity (%) 




Flispanic/Latino 

45.5 

48.1 

46.2 

Black, non-Hispanic/Latino 

52.6 

50.1 

52.3 

Receives housing assistance (%) 

55.9 

54.6 

44.2 

Flousing status (%) 




Rents 

83.5 

87.3 

86.3 

Lives with friends 

4.7 

3.4 

1.5 

Group shelter 

3.9 

1.6 

0.6 

No high school diploma or GED certificate (%) 

65.0 

52.0 

36.8 

Foreign-born (%) 

24.4 

30.2 

44.9 

Married or in a legal domestic partnership (%) 

13.5 

15.8 

36.7 

Working full time (%) 

27.5 

37.2 

65.1 

Average weekly earnings among those currently 




working ($) 

336 

377 

457 

Physical or mental health problem (%) 

30.3 

26.2 

10.6 

At risk of depression (%) 

16.3 

15.8 

9.2 

Sample size (total = 2,377) 

476 

1,427 

474 


(continued) 
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Table 2.6 (continued) 


Range of Earnings 


Bottom 20%, 

Middle 60%, 

Top 20%, 

Characteristic 

$0 - $3,375 

$3,376 -$13,125 

$13,126 -$32,820 

Pattern of reward receipt and coupon submissions 




Average number of automatically verified 
rewards earned 

17.7 

30.2 

43.8 

Number of coupon book rewards earned 

6.0 

31.3 

67.5 

Never submitted a coupon (%) 

35.9 

4.2 

0.0 

Number of activity periods family earned rewards 

10.4 

15.2 

17.5 

Sample size (total = 2,377) 

476 

1,427 

474 

Parents’ experiences with program (%) 




Payment processing decisions were 




Consistent over time 

71.7 

86.8 

88.0 

Prompt and efficient 

62.6 

73.0 

83.1 

Fair 

83.9 

91.3 

93.9 

Visited the Family Rewards Web site 

33.9 

54.0 

58.3 

Ever attended any workshop 

44.6 

51.3 

51.9 

Received a referral from NPO to find 




Education or training program 

27.4 

36.4 

42.3 

Job search program 

19.7 

31.8 

39.5 

Tutoring program 

28.2 

38.9 

41.7 

Received assistance from a friend or family 
member to claim rewards 

24.2 

25.9 

35.0 

Discussed Family Rewards with children when 
talking about school performance 

39.2 

41.4 

49.7 

Being in program 




Led to more family interactions 

42.2 

56.8 

64.0 

Got children thinking about family budget 

48.7 

55.2 

53.7 

Sample size (total = 715) 

121 

430 

164 


SOURCES: MDRC calculations using data from the Family Rewards 42 -month survey, Seedco's Family 
Rewards program data, and Baseline Information Forms. 

NOTES: The first program year covers September 2007 through August 2008; the second program year 
covers September 2008 through August 2009; and the third program year covers September 2009 through 
August 2010. 

GED = General Educational Development. NPO = Neighborhood Partner Organization. 

“This section excludes information for enrolled second parents in two-parent households. 
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were in the bottom 20 percent of earners in the first year; 209 (just less than half) had moved 
to a higher quintile in the last year of the program. This pattern suggests that families had a 
reasonable chance to improve their performance in the program over time (as measured by 
reward earnings), but that a small group was extremely difficult to reach. 

The Association Between Engagement with the Program and Reward Receipt 

It was noted earlier that the bottom 20 percent of earners relied most heavily on the au- 
tomatically verified public health insurance reward for their earnings from this program. This 
finding helps explain the pattern of reward receipt and submissions described in Table 2.6, 
which shows the characteristics of families who earned rewards for all three program years 
combined. On average, these families submitted only six coupons over the entire three years of 
the program, and 36 percent never submitted a coupon. Clearly, these families were not engaged 
by the program, or they did not understand or could not handle the coupon submission require- 
ments. 25 They showed a diminishing willingness to pick up their coupon books over time. In 
Year 1, 71 percent of the bottom 20 percent of earners picked up their coupon books (analysis 
not shown). In Years 2 and 3, about 60 percent picked up the book. 

Interestingly, over the three years, about the same percentage of low-earning families 
(24 percent) stated that they asked a friend or family member for help claiming rewards as 
families who earned a moderate amount (26 percent). This percentage is not very high in 
either case and may indicate an unmet need among some families for hands-on help from 
people they trust. 

The bottom 20 percent of earners did not seek much help from the program staff to get 
the additional support they may have needed. They were the least frequent users of the Web site 
and workshops, and they received the fewest referrals. They were the most dissatisfied with the 
payment processing system and they had the fewest interactions within their families that were 
prompted by the program. 

In contrast to the low-eamers, moderate-and high-eamers earned coupon-based and au- 
tomatically verified rewards more often, had a more positive view of the payment processing 
system, and made more use of the program services. The highest-earning families asked for the 
most referrals to educational and employment resources from NPO staff. A larger percentage of 
high-earning parents (35 percent) also received help navigating the program from family and 


25 A small number of families were never reengaged after they signed up for the study and were selected 
for the program group. These families never picked up coupon books. They may have moved, experienced a 
family trauma, or simply changed their mind about participating. It is also true that relocation or personal 
upheavals, such as a death in the family or a period of institutionalization, may have hampered participation at 
any time during the program. 


57 



friends. The highest-earning families attributed more changes in their interactions with their 
children to the program. 

It is not surprising that families who earned the most rewards described the program’s 
customer service most positively and felt that the program had the biggest impact on their 
family dynamics. This does not mean that engagement with the program caused the families to 
earn more money — the relationship between program engagement and reward earnings is not 
unidirectional (that is, earning money from the program is as likely to lead to more interactions 
with and discussions of the program as interacting with and discussing the program could lead 
to higher reward earnings) and it is strongly mediated by the participant’s ability to meet the 
conditions for a reward. At the same time, some findings from the 42 -month survey and the 
qualitative data about the behavior of higher earners suggest that cooperation between parents 
and children, parental oversight of children’s spending of their reward earnings, social support 
from friends and extended family, and supportive relationships with program staff were mecha- 
nisms by which the high-earning group may have increased their tendency to earn rewards. 
These factors characterize the ways foreign-bom families, particularly Spanish-speaking 
immigrant families, incorporated Family Rewards into their lives, and may be part of the reason 
this group was disproportionately represented among the highest earners in the sample. 


Ways Families Used the Rewards 

Parents were asked on the 18-month and 42 -month surveys to describe how they spent money 
that they earned from Family Rewards. Their spending habits provide some insight into how 
central the reward payments were to the household budget toward the end of the program. 

The main difference between parents’ uses of Family Rewards earnings from the first 
survey wave to the second one is that on the 42 -month survey parents reported fewer uses of the 
rewards; they made more strategic investments of their lower Year 3 earnings rather than 
spreading their spending across multiple categories of use. This can be seen in Table 2.7, where 
the two most common uses of the rewards at 42 months are household expenses such as rent 
and utilities (53 percent) and paying for a few “extras” such as eating out (62 percent). These 
were also the top two responses on the 18-month survey but were more commonly reported 
than on the 42 -month survey. At 18 months, 70 percent of parents stated that they were using 
reward money for household expenses, and 72 percent were using it for extras. 

Parents were asked if they were spending Family Rewards earnings on education or 
training for the first time on the 42 -month survey. Eleven percent indicated that they were. 

Additional analyses (not shown) demonstrate that uses varied predictably for families 
who earned the most from the intervention. Families who earned more money from the program 
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The Opportunity NYC Demonstration: Family Rewards 

Table 2.7 


Parents’ Report of Family’s Use of Rewards Received 
at 18 Months and 42 Months from the Time of Random Assignment 


Outcome 

Program Group 
18 Months 42 Months 

Use Family Rewards payments to (%) 

Flelp pay for regular expenses, such as rent, utilities, or food 

69.9 

52.6 

Pay off bills, such as credit cards or medical bills 

32.4 

22.1 

Make a major purchase, such as a house, major appliance, or car 

11.6 

8.0 

Save for some future need, such as college tuition or retirement 

46.0 

31.8 

Pay for health or dental care or health insurance 

15.2 

4.8 

Pay for things to help children in school, such as special lessons or 
private schools 

34.6 

21.7 

Pay for a few extras, such as eating out, going to a movie, buying 
electronics or clothes 

72.3 

62.2 

Flelp other family members or friends with expenses 

11.7 

8.9 

Pay for college tuition or other costs for training 

- 

11.2 

Other 

18.7 

4.7 

Sample size 

1,032 

717 


SOURCE: MDRC calculations using data from the Family Rewards 18-month survey and 
the Family Rewards 42 -month survey. 

NOTES: This table excludes control group members because it pertains only to the Family 
Rewards program. A randomly selected subsample of program group members was asked 
these questions. 

Sample sizes may vary because of missing values. 

A double dash (— ) indicates "not applicable." 

Percentages may sum to more than the number participating in any activity because 
sample members could list more than one response. 
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were more likely to state on the 42 -month survey that they were saving for some future need, 
such as college or retirement, and paying for special lessons to help children in school. 


Post-Program Aspirations and Plans 

The theory behind the conditional cash transfer program is that by providing families with 
additional financial resources in exchange for completing activities that promote health, 
educational achievement, and workforce participation, families will benefit immediately 
through a reduction in poverty over the short term and, over the long run, from the habits and 
skills that the incentives reinforced and the possible financial cushion created by any savings. 
The expectation — or hope — is that, in the absence of the financial incentives, participants 
should continue to engage in activities that had become routine during the program years and 
that were reinforced by mechanisms other than the financial incentives alone, and when their 
environment facilitated those behaviors. 

The evaluation explored whether participants hoped to sustain any of the positive be- 
havioral and lifestyle changes that they perceived they had achieved with the help of the 
program. The 42-month survey asked respondents how they planned to cope with the loss of 
reward income. Although about half of the survey respondents indicated that they intended to 
save their reward earnings to make them last longer, a sizeable proportion — 37 percent — said 
they would not be affected by the loss of income or would not do anything to replace it, as 
Table 2.8 indicates. About half of families reported a combination of strategies to cope with the 
loss of income. The most frequently reported combination was saving along with going back to 
school or working more. 

During in-depth interviews that were conducted just after the program ended, parents 
shared stories that represented the opposing perspectives outlined above. On the one hand, some 
parents expressed a desire to increase their workforce participation in order to sustain the 
standard of living that their families had enjoyed during Family Rewards. One mother explained 
how her job search became more urgent as the program drew to a close: 

I’ve been looking [for a job], but it just gave me more — 1 think after reality re- 
ally hit, then 1 got on it, like one day I’d get up and say, “Uh-oh.” Like, in 
March, 1 said, “Oh man — we’ve only got, like. . .” They sent you letters saying 
three more, two more payments, and I was, like, “Oh, wow.” And 1 started look- 
ing more in the computer. 1 was up late doing that. 

Another mother described how the benefit to her daughters from the extra income mo- 
tivated her: “[I wanted to] work as high as I can in the health field to make sure they can live 
well and be as much as possible and as wealthy as possible.” 
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The Opportunity NYC Demonstration: Family Rewards 

Table 2.8 

Families’ Plans to Make Up for Loss of Family Rewards Income 

Program 


Outcome Group 


Plans (%) 

Pick up more paid work (parent) 24.5 

Pick up more paid work (children) 11.5 

Go back to school 31.1 

Save money to make it last longer 54.0 

Take on more debt 10.2 

Apply to public assistance, such as food stamps or TANF 21.9 

Ask friends or family members for financial help 1 1 .2 

Other plans 1.7 

No plans 12.8 

Loss of rewards income unlikely to affect family 24.3 

Extent of planning 

Family listed 2 or more ways to make up for Family Rewards income loss (%) 52.5 

Average number of ways families plan to make up for Family Rewards income loss 2.0 


Sample size 717 


SOURCE: MDRC calculations using data from the Family Rewards 42-month survey. 

NOTES: This table excludes control group members because it pertains only to the 
Family Rewards program. A randomly selected subsample of program group members 
was asked these questions. 

Sample sizes may vary because of missing values. 

Percentages may sum to more than 100 percent because sample members could list 
more than one response. 


These mothers had high aspirations, but it is important to recognize that, in both cases, 
they were pursuing their job search on their own. Family Rewards retained a skeleton staff at 
the NPOs for seven months after the end of the third program year who could help families with 
paperwork and referrals to services related to education, health, and employment; however, 
parents did not seem to be aware of their availability or seek their help. 

On the other hand, some parents expressed utter resignation about the loss of income. A 
mother from the Bronx said plainly that she could not even imagine how to replace that money. 
“No,” she said, “I don’t think that... I haven’t thought about it. What am I going to do? The 
necessities will never go away. Things get more expensive every day.” Another mother said she 
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had no way to make a sustainable change in her income level because her job as a hair stylist 
was unpredictable — some days are good, others are bad, and that is the way it is. In some 
cases, parents thought their adolescent children could get summer jobs to soften the transition. 

Another dimension of the transition to post-program life was the need for parents to 
motivate their children to do well in school without being able to leverage the financial incen- 
tives. Many parents tried to motivate their children with words of encouragement or disap- 
pointment. These strategies met with varying levels of success. One such parent, who felt that 
her high school-age children’s academic perfonnance did improve, tried to encourage them to 
retain that level of motivation after the program ended by telling them: 

Through the program, you were taught to feel good about good grades. It was a 
learning experience. After you’ve worked together and come to a higher level, 
it’s not good to go back down. Because you see the good results. Aside from the 
incentives, you see the good results. 

But this parent had another child who had just started high school when the program 
ended, and she worried because this child was not willing to attempt as many Regents exams as 
her older siblings had been when they were eligible for the rewards. The parent said she was not 
sure how to overcome her daughter’s resistance in the absence of the rewards offer. 

With respect to health-related appointments, parents expressed a desire to be as diligent 
in setting and keeping appointments after the program as they had been when they had both the 
financial incentives and the dates on the coupons as reminders about when to set appointments. 
One parent found she had to fight the urge to procrastinate: 

Being in the program, it would. . .keep me — okay, yeah, you got to put this on 
the calendar. You got to schedule this in order for them to get their appointments 
and to complete the fomis that you have to fill out. ’Cause. . .right now, after the 
program, I know they have to have their appointments, but it’s like, okay, but I 
have to do this first and this and there. I keep pushing everything back, and I’m, 
like, “Oh, my gosh. I have to be on top of [these appointments] like before.” 

In general, parents were ambivalent about their lives after the program ended. They 
frequently expressed gratitude for having had the opportunity to participate in the program 
and a desire to sustain perceived positive changes. However, their optimism about the future 
was constrained by limited financial resources, the burdens of daily life as mostly single 
parents raising children in poverty, and, in some cases, their difficulty imagining an economi- 
cally stable life. 
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Conclusion 


The Family Rewards experiment is one of many programs that have used financial incentives to 
encourage particular health, educational, and workforce behaviors. It was more complex than 
other financial incentive programs because it targeted two generations within the family across 
these three domains. This chapter demonstrates that Family Rewards succeeded as a vehicle to 
transfer cash to families in exchange for completing certain activities. The payment processing 
and marketing systems were sound, and most families earned a significant annual cash transfer, 
particularly for activities related to health and education. Participating adults and their children 
described the financial incentives as generally aligned with their own values, encouraging 
greater follow-through or engagement with positive behaviors, but not so powerful that they 
motivated participants to abandon their values or chosen priorities just so they could earn extra 
money, especially when it came to work. In this regard, the program designers were successful 
in developing a program involving cash incentives for low-income people that was not coercive 
and did not, according to qualitative and survey evidence, generate high levels of conflict within 
families. 26 However, participants clearly expressed an unmet need for more support than the 
“incentives only” model allowed to actually achieve the outcomes for which they were striving. 
In making the transition to post-program life, families also required guidance for sustaining or 
amplifying any gains they had made while they were receiving the rewards. The next generation 
of the Family Rewards program, described in Chapter 7, attempts to address those gaps. 


26 Though not reported in this chapter, the survey and qualitative research in the core Family Rewards 
evaluation, and a supplemental study of high school students and parents (Morris, Aber, Wolf, and Berg, 
2012), explored the issue of family conflict. One of the concerns about offering incentives to families — and 
paying some of the reward money directly to high school students — was that it might increase conflict 
between intimate partners or between parents and their teenage children over the use of the money. The 
qualitative and survey research found that families who received the incentives had more conversations about 
money, but families did not describe higher levels of conflict than they had experienced before the program 
began or compared with the control group’s experience. 


63 




Chapter 3 

Effects on Income and Material Well-Being 


The long-term goal of Family Rewards is to reduce poverty across generations by promoting 
human capital development among low-income children and their parents. The short-term 
goal is to reduce current poverty and material hardship through its direct cash reward pay- 
ments — or conditional cash transfers (CCTs) — and through any immediate increases it 
could generate in parents’ earnings from employment, which could also boost Earned Income 
Tax Credit (EITC) payments. The early evaluation results showed that Family Rewards 
succeeded in substantially improving families’ economic circumstances in its first two years, 
and the extra cash rewards received from the program helped families reduce a variety of 
material hardships. 1 Chapter 2 in the current report shows that families continued to earn cash 
transfers through the end of the program, with the majority of them earning substantial cash 
accounts. 

This chapter examines the effects of the cash transfers on a broad range of outcomes 
related to income and well-being. It considers how, and to what extent, participation in 
Family Rewards continued to alleviate poverty and hardship. It looks at the program’s effect 
on these and other material circumstances while the cash transfers were in effect, and pro- 
vides an early glimpse of these effects after the cash payments ended. The condition of the 
economy was worsening during the early part of the evaluation period, and thus the chapter 
examines the program’s effects on those outcomes in a tough economic context. 

These analyses draw heavily on the Family Rewards 42-month parent survey, as well 
as administrative records data on Temporary Assistance for Needy Families (TANF) and food 
stamp receipt provided by the New York City Human Resources Administration (HRA). The 
42-month survey was administered to a random subset of the full study sample, and provides 
extensive data on 2,966 families. 2 Most of these interviews were conducted between Novem- 
ber 2010 and June 2011, after the three-year program had ended in August 2010. Thus, all 
interviews were completed after respondents could earn any more rewards — and almost all 
(91.6 percent) were interviewed after they had received their final payment. With the excep- 


See Chapter 5 of Riccio et al. (2010) for early evaluation findings on income and well-being. 

2 The 42-month survey provides information about Family Rewards sample members on a broad set of 
topics such as participation in employment and education activities, health care, employment and job character- 
istics, household composition, and child outcomes. Overall, 2,966 sample members completed the survey 
interview, resulting in a response rate of 79 percent. Appendix J, which appears in the supplement to this report 
(Riccio et al., 2013, available at www.mdrc.org), provides additional details on the survey effort and analyzes 
response patterns. 
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tion of some retrospective measures, the 42-month survey largely provides a snapshot of 
families’ circumstances after the program had ended — also referred to as the “early post- 
program” period. 

Box 3.1 explains how to interpret the tables that show estimated program impacts that 
are presented throughout the remainder of this report. These tables cover a large number of 
impact estimates that are relevant to family poverty, hardship, and economic security in the 
three domains in which rewards are provided: children’s education, family preventive health 
care, and parents’ work and training. The estimates of program impacts were calculated 
controlling for a range of pre-random assignment background characteristics, such as the 
parents’ race/ethnicity, education level, marital status, and employment status. As the number 
of outcomes that are examined increases, the probability of findings that are statistically 
significant simply by chance also increases. 3 Although the impact analysis does not formally 
account for “multiple hypothesis testing,” 4 caution is used when interpreting impacts that do 
not appear to be part of a larger pattern of impacts within a given set of measures. 

Overall, the results presented here show that when the program was operational and 
families were eligible to earn rewards, Family Rewards continued to achieve some of its 
immediate goals: it reduced the proportion of families who were living in poverty (and in 
severe poverty), reduced reports of material hardship, and improved parents’ sense of their 
family’s financial well-being. However, in the early post-program period, when most of the 
42 -month survey interviews were conducted and when families could no longer boost their 
income with reward payments, the program’s effects on household income and poverty fade. 
This pattern is also evident for a few other positive outcomes that were documented in the 
early stages of the evaluation. 5 Yet, despite the loss of substantial additional income from 
reward payments, the program families appear to report lower levels of some types of hard- 
ships (for instance, food insufficiency) in the post-program period than does the control 
group. However, those effects are most obvious for program families who were interviewed 
closer to the end of the program in August 2010, suggesting that those positive effects may be 
lingering effects of the cash transfers. 


3 Statistical significance indicates the extent to which the difference between the program and control 
group outcomes — or the “impact” of the program — is likely to have been a true result of the program. 

4 Multiple hypothesis testing is concerned with the idea that when a large number of impacts is examined, 
some will be statistically significant simply by chance. 

5 See Riccio et al. (2010). 
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Box 3.1 


How to Read the Impact Tables in This Report 

In the context of this evaluation, an “impact” is a measure of how much Family Rewards changed 
outcomes for program participants. All the tables in this report that show impacts use a similar 
format, illustrated in the table excerpt below, which presents data on two material hardship and 
well-being outcomes that were obtained from parents’ reports on the Family Rewards 42-month 
survey. The top row, for example, shows that 5.5 percent of respondents in the program group had 
their utilities turned off in the 12 months before the survey interview, compared with 8.0 percent 
of control group respondents. 

Because families were assigned randomly either to the program group or to the control group, the 
effects of the program can be estimated by the difference in outcomes between the two groups. 
The “Difference” column in the table shows the differences between the two research groups’ 
outcomes — that is, the program’s estimated impacts on the outcomes. For example, the estimated 
program impact on having a utility turned off can be calculated by subtracting 5.5 percent from 
8.0 percent, yielding a reduction, or estimated impact, of -2.5 percentage points. 

The p-value shows the probability that this difference, or impact, arose by chance. In the table 
excerpt below, the difference between the program and control groups in having a utility turned 
off has less than a 3 percent probability of arising as a result of chance rather than as a result of 
the Family Rewards program. In contrast, the difference on the measure of financial well-being 
(bottom row) has less than a 2 percent probability of having arisen by chance. For this evaluation, 
only differences that have a 10 percent probability or less of arising by chance are considered 
“statistically significant” and therefore represent true program impacts. The number of asterisks 
indicates whether the impact is statistically significant at the 1 percent (***), 5 percent (**), or 10 
percent (*) level. An impact that is statistically significant at the 1 percent level, for example, has 
a 1 percent chance or lower of having arisen by chance. 

The effect size, shown in the final column, is the difference between the program and control 
group outcomes divided by the “standard deviation” of the outcome (a measure of its variability). 
Expressing an impact in standard deviation units helps to interpret its size, particularly when the 
outcome is in nonstandard units, such as scale scores, as in the second row. The interpretation of 
effect sizes has typically followed guidelines that consider an effect size of 0.20 to be small, 0.50 
to be medium, and 0.80 to be large. 1 Others argue that effect sizes should be interpreted in the 
context of the outcome and demonstration. 1 

Impacts on Two Material Hardship and Well-Being Outcomes, 
from the Family Rewards 42-Month Survey 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- 

Value 

Effect 

Size 

Utility (gas, oil, electricity) turned 
off in past 12 months (%) 

5.5 

8.0 

-2.5** 

0.028 

— 

Financial well-being scale 
(4 = low; 16 = high) 

9.0 

8.8 

0.2** 

0.017 

0.108 


NOTE: Effect size is not calculated for discrete variables, 
t Cohen (1988). 

JSee, for example, Hill, Bloom, Black, and Lipsey (2008). 
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Income and Poverty 

A central goal of the Family Rewards cash incentives was to boost household income and 
reduce immediate poverty and hardship. To characterize household income and poverty, 
families completing the 42 -month survey were asked to list their sources of income and total 
household income for the calendar month prior to the month in which their survey interview 
was conducted. 6 As the majority of families were interviewed after the program had ended, two 
sets of calculations are presented here. One set captures income and poverty at the time of the 
42 -month survey interview — essentially reflecting income and circumstances in the early post- 
program period. 7 A second set of estimates pertains to income and poverty in Year 3 (the last 
year of the program, from September 2009 to August 2010) and includes reward payments 
earned by participants in that year. 8 Both of those estimates are presented in Table 3.1. 

As the first panel of Table 3.1 shows, the estimated average monthly household income 
in Year 3 was $1,973 for the program group (including reward payments) and $1,620 for the 
control group, representing a statistically significant gain of $353 per month (or about a 22 
percent increase relative to the control group’s income). On average, families in the program 
group earned an additional $276 each month in reward payments during the three years of the 
program. 9 Excluding those rewards in the early post-program measure of income shows that the 
estimated average monthly household income at the time of the interview was $1,700 for the 
program group and $1,620 for the control group, representing a statistically significant but 
smaller gain of about $79 per month (or about 5 percent of the gain with reward payments 
excluded). To a large extent, the income gains demonstrated during the program are mainly 


Respondents were instructed to exclude tax refunds and program reward payments, and to include in- 
come from all other sources such as their job(s), jobs of other household members, food stamps, child support, 
TANF, Supplemental Security Income (SSI), and unemployment insurance, among other sources, for everyone 
living together in the household. Respondents were also asked to exclude EITC payments because of the 
difficulty of converting it into a monthly amount. The reported sources of income were combined and used to 
calculate household poverty and income relative to the federal poverty threshold, using the 2010 and 2011 
poverty guidelines, depending on when a respondent was interviewed. (In 2010, the federal poverty level for a 
family of three was $18,310.) Because the poverty estimates presented in this report have a particular defini- 
tion, readers should be cautious about comparing them with estimates from other published sources. 

Roughly 76 percent completed this interview within six months from the time when the program ended. 
Thus, for the most part, the survey captures circumstances early in the post-program period. 

8 The estimate of household income from the 42 -month survey closely matches the estimate obtained from 
the 18-month survey, which was conducted when the program was in effect and provides an estimate of 
income and poverty during the program period. 

Rear 3 payment data, which cover August 2009 to September 2010 and overlap with the survey period, 
are used to estimate the average monthly incentive payment during this period. The average monthly incentive 
amount of $276 that is reported in the table is about 17 percent of the control group income. More information 
on families’ participation and reward receipt patterns is offered in Chapter 2. 
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Table 3.1 


Impacts on Income and Income Sources 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Income and poverty (Year 3) 

Average total household income in month prior to 
interview (including Family Rewards payments) a ' D ' c ($) 

1,973 

1,620 

353 *** 

0.000 

Average monthly Family Rewards payment, Year 3 l1 ($) 

276 

- 

- 

- 

Flousehold income at or below the federal poverty 
level (including rewards) a ’° (%) 

56.0 

68.2 

- 12.2 *** 

0.000 

Total household income in prior year as a percentage 
of the federal poverty level (including rewards) a D (%) 
Less than 50% 

16.3 

27.4 


0.000 

50% - 100% 

39.8 

40.8 

- 1.0 

0.655 

101% - 129% 

15.8 

11.2 

4 ~j *** 

0.004 

130% or more 

28.1 

20.6 

r j ^ *** 

0.000 

Income and poverty (earlv post-program) 

Average total household income in month prior to 
interview (excluding Family Rewards payments) a ' e ($) 

1,700 

1,620 

79 * 

0.093 

Flousehold income at or below the federal poverty 
level (excluding rewards) a,e (%) 

66.2 

68.2 

- 2.0 

0.309 

Total household income in prior year as a percentage 
of the federal poverty level (excluding rewards) a e (%) 
Less than 50% 

25.9 

27.4 

-1.5 

0.499 

50% - 100% 

40.3 

40.8 

- 0.6 

0.754 

101%- 129% 

13.2 

11.2 

2.1 

0.167 

130% or more 

20.6 

20.6 

- 0.1 

0.976 

Income sources (%) 

f 

Flousehold income source in month prior to interview 
Respondent's earnings 

59.6 

50.3 

9 3 *** 

0.000 

Other household members' earnings 

23.2 

20.7 

2.5 

0.163 

Food stamps 

69.6 

70.7 

- 1.1 

0.554 

Child support 

18.7 

18.2 

0.5 

0.754 

Temporary Assistance for Needy Families (TANF) 
or other cash assistance 

14.5 

16.1 

- 1.6 

0.292 

Supplemental Security Income or Disability 

28.5 

32.4 

_3 9 ** 

0.042 

Unemployment insurance (UI) 

9.8 

8.9 

0.8 

0.522 

Other 

73.5 

72.6 

0.9 

0.655 

Taxes (%) 

Filed for taxes in prior year and used tax 
preparation service 8 

55.8 

54.8 

1.0 

0.628 

Filed for taxes in prior year and received a refund- 
anticipation loan 

13.7 

13.2 

0.5 

0.767 


(continued) 
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Table 3.1 (continued) 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Ever employed since random assignment 
Filed for taxes in prior year and used tax 
preparation service 8 

65.6 

66.5 



Filed for taxes in prior year and received a re fund- 
anticipation loan 

16.7 

17.0 

— 

— 

Sample size (total = 1,982) 

1,024 

958 




SOURCE: MDRC calculations using data from the Family Rewards 42-month survey. 

NOTES: Sample sizes may vary because of missing values. 

The items in this section of the survey were administered to a random subsample (N = 1,982) of the 
survey respondents. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the difference between the program and control groups 
arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; 
* = 10 percent. 

Italic type indicates comparisons that are nonexperimental. Statistical tests were not performed. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

A double dash (--) indicates "not applicable." 

“Monthly household income amounts equal to or greater than $10,000 were excluded from this 
calculation. About 7.2 percent of the sample is excluded from the income measures because respondents 
did not know or refused to provide the information. An additional 0.6 percent of the sample was 
excluded because the income provided was over $10,000. 

b Annual household income is calculated by multiplying by 12 the respondent's income in the month 
prior to the survey interview. For program group members, it includes Family Rewards payments earned 
during program Year 3. The federal poverty level was calculated based on annual income (monthly 
income multiplied by 12) and the household size at the time of the survey. The poverty threshold was 
measured according to the 2010 or 201 1 Poverty Guidelines, depending on when a respondent was 
interviewed. 

c The Year 1 income measures reported on the 18-month survey are within 3 percent of the 42- 
month income measures reported here. 

d Family Rewards payments are based on Seedco's Family Rewards data from program Year 3, 
which includes activities completed in September 2009 through August 2010. The monthly Family 
Rewards payment amount is calculated by dividing the annual reward amount by 12. The payment data 
do not include bonus payments that some families received for opening new bank accounts. 

e Annual household income is calculated by multiplying by 12 the respondent's income in the month 
prior to the survey interview. This calculation does not include Family Rewards payments earned during 
program Year 3. The federal poverty level was calculated based on annual income (monthly income 
multiplied by 12) and the household size at the time of the survey. The poverty threshold was measured 
according to the 2010 or 201 1 Poverty Guidelines, depending on when a respondent was interviewed. 

f Percentages may add up to more than 100 percent because respondents may have multiple income 
sources. 

BThis includes free tax preparation services and paid tax preparers. 
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driven by the cash transfers that the families received over the three years of their participation 
in Family Rewards. The small, early post-program impact on income is potentially driven by 
Family Rewards’ effect on employment; as discussed in more detail in Chapter 6, the program 
led to a small increase in survey-reported employment (most likely in jobs not covered by the 
unemployment insurance, or UI, system). 

Next, poverty rates are calculated by estimating annual family income relative to the 
federal poverty level pertaining to each family’s size. 10 Table 3.1 presents the distribution of this 
measure of poverty across four levels, including severe poverty, which is defined as income 
below 50 percent of the poverty level. The Year 3 estimate, which includes reward payments in 
the calculation of average household income, shows that Family Rewards substantially altered 
the distribution of families across different levels of poverty, especially by reducing the propor- 
tion of households in severe poverty and increasing the proportion with income at or above 130 
percent of the federal poverty level. For example, while 27.4 percent of the control group had 
income falling into the severe poverty category, the rate for the program group was 16.3 
percent, representing a statistically significant drop of 11.1 percentage points. Similarly, 28.1 
percent of the program group had income at or above 130 percent of the federal poverty level 
compared with 20.6 percent of the control group, a statistically significant gain of 7.5 percent- 
age points. This poverty reduction effect of the program is generally expected, especially given 
the substantial cash transfers earned by the majority of families. 

In the next panel (“Income and poverty”), the early post-program estimate of poverty, 
which does not include the additional income earned from reward payments, shows that once 
the reward payments stopped at the end of Year 3, the program’s poverty reduction effect faded. 
For example, 68.2 percent of the control group had income at or below the federal poverty level, 
and the rate for the program group was 66.2 percent, a 2.0 percentage point difference that is not 
statistically significant. The reductions in poverty in the final year of the program were mostly 
driven by the cash transfers themselves — and not by changes in other sources of income — 
and once those transfers ended, the income and poverty effects were not sustained. 

To gather information about the program’s effects on sources of income, the 42-month 
survey contained a question asking whether respondents had received income or benefits in the 
prior month from a variety of sources, including earnings, government benefits, and child 
support. The third panel of Table 3.1 (“Income sources”) shows that Family Rewards neither 
increased nor decreased the likelihood of receiving income from most of those sources. Howev- 
er, it did appear to increase earnings from employment and slightly decrease the likelihood of 
receiving Supplemental Security Income (SSI) or disability assistance. The observed earnings 


"'Annual household income is estimated based on the one-month household income snapshot gathered at 
the time of the survey interview. 
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and employment effects are discussed more fully in Chapter 6. It is less clear why Family 
Rewards would reduce SSI and disability assistance receipt for the program group families. 
This issue may require further investigation, but there was a small difference in SSI receipt at 
study enrollment, suggesting that this result is not an effect of the program. 11 

Administrative records data were used to estimate the program’s effects on cash aid from 
the TANF and Safety Net Assistance (SNA) programs and food stamp benefits. 12 The available 
data covered three years of follow-up after the quarter of random assignment. Overall, the results, 
presented in Table 3.2, show that Family Rewards had no effect on benefits receipt — either the 
likelihood of receiving payments or the average dollar amount received. Among program group 
members, 76.1 percent received food stamp benefits at some point (that is, not continuously) 
over the three-year follow-up period, compared with 76.5 percent of the control group. Similarly, 
roughly the same proportions of program and control group families ever received TANF/SNA 
assistance during the follow-up period (43.4 percent versus 43.8 percent in the program and 
control groups, respectively). 


Banking and Financial Services 

Family Rewards made payments electronically to families who earned rewards, depositing the 
money into bank accounts or onto stored value cards. 13 The program provided all parents and 
high school students with a chance to use special savings accounts that, with the cooperation of 
several banks and credit unions, were designed explicitly for the program. Called “Opportunity 
NYC accounts,” these special accounts did not carry fees or minimum balances and could not 
be overdrawn. 14 Over the three years of the program, the organizations operating Family 
Rewards strongly urged all families to maintain their bank accounts. 

The bank accounts provided the program with an efficient mechanism for paying out 
cash rewards. In addition, this linkage with mainstream financial institutions had the potential to 
spur other positive effects for families’ financial security. For one, the designers of Family 
Rewards recognized the possibility that with these accounts and the extra money earned from 


1 'information about SSI receipt at study entry was collected as part of the 42 -month follow-up survey. 

12 The SNA program provides assistance to individuals and families in New York State who do not qualify 
for the time-limited federal TANF program. SNA payments may take the form of direct cash aid to beneficiar- 
ies or vendor payments (for example, to landlords) made on their behalf. 

13 Only about 11 percent of the participants opted for stored value cards. Program operations data show 
that the majority of families had bank accounts linked to the program for payment purposes. 

14 In the early phases of the program, recognizing that families might also benefit from some guidance on 
how to manage the money they earned from the program, Seedco and the Neighborhood Partner Organizations 
provided them with information (including a special workshop) on the basics of budgeting, money manage- 
ment, and debt management, all issues related to asset-building. 
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Table 3.2 

Impacts on Temporary Assistance for Needy Families (TANF) or Safety Net 
Assistance (SNA) and Food Stamp Receipt and Payments, Years 1 to 3 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Ever received TANF/SNA (%) 

Year 1-3 

43.4 

43.8 

-0.5 

0.627 

Year 1 

37.2 

38.1 

-1.0 

0.281 

Year 2 

34.4 

34.0 

0.4 

0.691 

Year 3 

32.1 

30.9 

1.1 

0.295 

Amount of TANF/SNA received (S') 

Year 1-3 

6,236 

6,066 

170 

0.477 

Year 1 

2,238 

2,140 

98 

0.244 

Year 2 

2,081 

2,018 

63 

0.479 

Year 3 

1,916 

1,907 

9 

0.924 

Ever received food stamps (%) 

Year 1-3 

76.1 

76.5 

-0.4 

0.701 

Year 1 

66.9 

67.5 

-0.6 

0.598 

Year 2 

68.3 

68.9 

-0.6 

0.578 

Year 3 

69.3 

70.3 

-1.0 

0.371 

Amount of food stamps received ($) 

Year 1-3 

9,283 

9,354 

-71 

0.717 

Year 1 

2,479 

2,483 

-4 

0.942 

Year 2 

3,286 

3,329 

-43 

0.572 

Year 3 

3,517 

3,541 

-24 

0.763 

Sample size (total = 4,749) 

2,377 

2,372 




SOURCE: MDRC calculations using administrative records data from the New York State Human 
Resources Administration. 


NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the difference between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. Standard errors were adjusted to account for multiple 
observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Dollar averages include zero values for sample members who were not receiving TANF/Safety 
Net Assistance or food stamps. 


73 






the program, some families would save and try to build assets. Second, by connecting families 
to mainstream banking institutions, the program also potentially steered families away from 
using alternative services like check cashers or “payday loans,” 15 which are often available to 
low-income households at very high prices and in ways that many experts contend mislead and 
exploit the poor. 16 

Local and national studies show that low-income populations are less likely to hold 
bank accounts and more likely to face high costs for basic financial transactions through non- 
mainstream financial institutions. 17 The use of check cashers and payday loans may reflect 
lower financial literacy, as users accept higher interest rates than those offered by mainstream 
banks, or it may reflect limited access to mainstream financial institutions (because of hours or 
forms of discrimination) and short-term credit available through credit cards. For Family 
Rewards, about 53 percent of the sample did not have a bank account at the time of program 
enrollment. 18 Also, early evaluation findings show that, at least in the short term, Family 
Rewards was effective in linking program group families to mainstream banking institutions 
and reducing their reliance on alternative options like check cashers and payday loans. 

The 42 -month survey provides a longer-term assessment of these effects. The data show 
that Family Rewards continued to have a positive effect on the banking behaviors of program 
participants: roughly four years into the follow-up period, and after Family Rewards had ended, 
the program group was more likely to remain connected to mainstream financial institutions. As 
shown in Table 3.3, 64 percent of the program group reported having a bank account at the time 
of this interview, compared with 46.6 percent of the control group respondents, a statistically 
significant difference of 17.5 percentage points. 19 The program group was also more likely (by 
12.0 percentage points) to report having a checking account. Although similar proportions of the 
program and control groups were likely to close a bank account at some point following random 
assigmnent (10.7 percent and 11.0 percent, respectively), the program clearly generated a net 
increase in the proportion who had a bank account, even after the program had ended. 


,5 A payday loan is a loan from a check-cashing outlet or other lending institution that must be repaid by 
the next payday. Payday loans are illegal in New York, but may be available through the Internet. 

16 Barr and Blank (2009). 

17 Barr (2004). 

18 See New York City Department of Consumer Affairs (2008) for more information on the extent to 
which families are unbanked (that is, do not have a bank account) in different parts of New York City. 

19 From Chapter 2, it is clear that a much higher percentage of program group families earned a reward in 
the final year of the program and had deposits made to their bank account than is reported in the 42 -month 
survey. After the program ended, it is possible that some families with active bank accounts were no longer 
using them and failed to recall having a bank account when interviewed. A smaller number of other partici- 
pants received their payments via stored valued cards. 
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Table 3.3 


Impacts on Banking, Savings, and Debt 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Use of banking/financial services (%) 

Currently has any bank account 

64.0 

46.6 

27 5 H=H=H= 

0.000 

Currently has checking account 

55.3 

43.3 

12.0 *** 

0.000 

Had a bank account closed last year 

10.7 

11.0 

-0.3 

0.844 

Financial transactions at least once a month 

Cash check at check casher 

29.2 

31.5 

-2.3 

0.260 

Pay bill at check casher 

36.8 

39.1 

-2.3 

0.288 

Use ATM card to access cash 

64.7 

54.9 

9 g H=H=H= 

0.000 

Take cash advance on credit card 

4.0 

4.1 

-0.2 

0.863 

Bounce check or overdraw checking account 

4.9 

4.1 

0.7 

0.432 

Get payday loan 

1.0 

1.2 

-0.2 

0.710 

Family savings and debt 

Average savings 3 ($) 

440 

361 

80 

0.314 

$0 (%) 

75.4 

83.2 

j~! g *** 

0.000 

$1 - $250 (%) 

6.9 

3.6 

3 3 *** 

0.002 

$251 -$500 (%) 

5.3 

3.9 

1.3 

0.176 

More than $500 (%) 

12.5 

9.2 

3 2** 

0.023 

Any (%) 

24.6 

16.8 

y g *** 

0.000 

Average debt* 5 ($) 

6,425 

6,121 

304 

0.544 

$0 (%) 

42.0 

41.9 

0.2 

0.932 

$1 -$1,000 (%) 

6.0 

6.8 

-0.8 

0.489 

$1,001 -$5,000 (%) 

23.5 

21.2 

2.3 

0.238 

$5,001 -$15,000 (%) 

17.1 

18.6 

-1.6 

0.378 

More than $15,000 (%) 

11.4 

11.6 

-0.1 

0.929 

Outstanding loans, bills, or payments (%) 

None 

41.5 

42.1 

-0.6 

0.782 

Car loan 

6.4 

5.3 

1.1 

0.300 

Home loan 

2.9 

1.1 

2 9 *** 

0.003 

Back rent 

9.9 

13.0 

-3.1 ** 

0.033 

Student loan 

19.7 

21.0 

-1.4 

0.446 

Hospital or medical bill 

18.0 

17.3 

0.7 

0.672 

Credit card or store bill 

44.9 

42.7 

2.2 

0.316 

Child support payments 

0.4 

0.2 

0.2 

0.395 

Other 

4.3 

5.2 

-0.9 

0.339 


(continued) 
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Table 3.3 (continued) 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Currently repaying (%) 





Any loan 

42.0 

39.7 

2.3 

0.291 

Car loan 

5.5 

3.6 

2.0 ** 

0.036 

Flome loan 

2.7 

1.0 

j y *** 

0.005 

Student loans 

9.4 

8.1 

1.4 

0.280 

Flospital or medical bill 

5.0 

3.8 

1.2 

0.205 

Credit card or store bill 

30.3 

24.2 

6.2 *** 

0.002 

Other 

2.5 

2.9 

-0.4 

0.545 

Sample size (total = 1,982) 

1,024 

958 




SOURCE: MDRC calculations using data from the Family Rewards 42-month survey. 


NOTES: Sample sizes may vary because of missing values. 

The items in this section of the survey were administered to a random subsample (N = 
1,982) of the survey respondents. 

A two-tailed t-test was applied to differences between outcomes for the program and 
control groups. The p-value indicates the likelihood that the differences between outcomes for 
the program and control groups arose by chance. Statistical significance levels are indicated as 
follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 
a About 6 percent of the sample is excluded from this analysis because of missing data. 
b About 3.6 percent of the sample is excluded from the debt measures because respondents 
did not know or refused to provide the information. An additional 0.2 percent of the sample was 
excluded because the debt reported was $100,000 or more. Debt amounts equal to or greater 
than $ 1 00,000 were excluded from these calculations. The survey questions on savings and debt 
are largely framed around family finances; thus, it is mostly likely that participants are reporting 
debt accumulated by the family. 


Table 3.3 also presents the proportion of sample members reporting use of alternative 
financial services. The survey asked whether they had used these fringe financial services 
several times a month, about once a month, every few months, a few times during the year, or 
never (not shown specifically in Table 3.3). Program group members were less likely to report 
on the 42-month survey that they had used these types of services. However, but for the excep- 
tion of one, most of these differences are not statistically significant. As expected, Family 
Rewards continued to increase the use of ATMs for cash withdrawal, a benefit made possible 
by having fomial banking accounts. Not surprisingly, very few sample members reported 
getting a payday loan, which, as noted above, is illegal in New York State. 
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Savings and Debt 

Participants in Family Rewards could use their program reward payments as they pleased — in 
other words, no restrictions were placed on how the rewards should be used. The program 
designers believed that imposing restrictions (such as requiring that the reward money be used 
only for education, training, or home ownership, as some asset-building programs do, or for any 
particular purpose) would have reduced the incentive value of the rewards. Thus, one question 
is whether, without such restrictions, the families would still use the additional income from the 
program to build their financial security — a positive program effect that was evident in the 
early evaluation findings. Previous studies suggest that resources received in large, lump sums, 
rather than in smaller, more regular payments, are more likely to be saved, used to pay down 
debt, or invested in human capital. 20 

At the 42-month point, roughly 25 percent of the program group reported having some 
family savings after the program ended, compared with 17 percent of the control group, a 
statistically significant difference of 7.8 percentage points. The program also increased the 
proportion who had more than $500 in savings by 3.2 percentage points. Some of these higher 
average household savings are concentrated among two-parent families and those who are less 
likely to be receiving government housing assistance, 21 suggesting that such families may have 
been better able to save than their single-parent or housing-assisted counterparts. 22 However, 
while the distribution of average household savings shows some positive effects, the program 
group and control group reported similar levels of average household savings overall — $440 
for the program group compared with $361 for the control group, a difference of about $80 that 
is not statistically significant, as shown in Table 3.3. 23 This is in contrast to the early evaluation 
finding that program group families were more likely to report savings than their control group 
counterparts ($575 for the program group, compared with $374 for the control group). Over 
time, average household savings reported by control group members held more or less constant, 


20 Chambers and Spencer (2008); Smeeding, Phillips, and O’Connor (2000). 

21 See supplementary Appendix Tables F.l to F.6 in Riccio et al. (2013), available at www.mdrc.org. 

22 Looking at predictors of the use of Family Rewards cash transfers, a recent analysis finds that house- 
holds demonstrating greater financial literacy, those with savings goals, and those who do not use alternative 
financial services are more likely to use some or all of their cash rewards to build savings or human capital. On 
the other hand, households demonstrating greater access to or use of mainstream banking services and credit 
are more likely to use some or all rewards to pay down debt. See Verma et al. (2011). 

23 The survey respondents were asked to report how much they (and their spouse or partner) had in sav- 
ings. (At enrollment, a majority of the participants — 80.9 percent — were in one-parent families). The 
reported savings could include any money or savings kept at home or elsewhere, such as money in a savings 
account, money market fund, credit union, pension fund, stocks or bonds, or certificates of deposit. The 
calculation shown in Table 3.3 includes those with zero savings, so for those with any savings, the amount 
could be substantially higher. 
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but savings reported by program group members declined, possibly a function of the loss of 
income from the cash rewards. 

Stemming from the broad question about families’ use of the additional income is the 
related question that asks whether the reward payments were used to clear debt. Those findings 
are also presented in Table 3.3. At the start of the program, it was unclear whether families 
would use the extra income from Family Rewards to manage their finances better — clear their 
debt, for example — or whether it would encourage them to incur additional debt, recognizing 
that an increase in debt is not necessarily always a sign of poor financial management. As 
documented in the early evaluation report, the 18 -month survey showed that the program 
reduced the proportion of program group families with debt by a statistically significant 3 . 1 
percentage points. 24 To explore this potential effect over the longer term, the 42 -month survey 
asked respondents to indicate whether they had any debt and, if they had, the amount owed. 25 
Table 3.3 shows that both groups reported relatively high amounts of family debt, but the 
difference is not statistically significant ($6,425 for the program group versus $6,121 for the 
control group). 26 Those families who reported higher levels of debt (over $5,000) were repaying 
multiple types of loans or bills, including credit card payments, student loans, or medical bills. 

Respondents were also asked about the types of loans or payments that were outstand- 
ing and the types that they were repaying at the time of the interview. Discussions with families 
over the course of the first three years of the program suggest that some were able to use their 
cash rewards for such purposes. The data from more recent interviews, particularly after the 
program had ended, provide a perspective on families’ coping strategies and the types of 
adjustments they made to meet their obligations and to sustain financial stability. This 
participant describes how she adjusted her budget to manage debt repayments while 
attempting to cover other needs: 

1 had to adjust my budget based on what it is that — the changes that was made 
in August.. . . Yes, where would I cut back? Where would my cutbacks be? My 
cutbacks would be on debtors that 1 was paying on a bimonthly basis. 1 had 
agreements that 1 made, 1 needed to contact them and let them know that unfor- 
tunately I wouldn’t be able to make those payments. We’d have to adjust and 
weigh the payments, so instead of me making $300 every two months, then now 
1 have to adjust it back based on my income. 


24 Riccio et al. (2010). 

25 The survey questions on savings and debt are largely framed around family finances, and it is likely that 
participants are reporting debt accumulated by the family rather than simply the respondents’ personal debt. 

26 The calculation of average debt excludes four cases reporting extremely high levels of debt (over 
$100,000). The cases were excluded to test the sensitivity of the results to the exclusion of a few extreme 
values. At least two of the four were paying back home loans. 
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The second panel (“Family savings and debt”) in Table 3.3 presents data on financial 
stability from the 42-month survey. As shown, respondents noted various types of outstanding 
loans and payments. For example, roughly 45 percent of the program group reported having an 
unpaid balance on a credit card or store bill, 27 compared with 43 percent of the control group, a 
difference that is not statistically significant. Eighteen percent of the program group reported 
having outstanding hospital or medical bills, compared with 17.3 percent of the control group 
(again, a difference that is not statistically significant). However, the cash incentives appear to 
have reduced the program group’s likelihood of having outstanding back rent — as shown, 
almost 10 percent of the program group reported this hardship, compared with 13 percent of the 
control group, a 3 percentage point reduction that is statistically significant. 

As shown at the bottom of Table 3.3, under “Currently repaying,” both program and 
control group members are almost equally likely to be paying off any loan (42 percent of the 
program group versus 39.7 percent of the control group), but the program group participants 
were more likely to be paying off certain types of loans: 30.3 percent of the program group 
members said they were paying off credit card or store bills, compared with 24.2 percent of 
the control group, a statistically significant increase of 6.2 percentage points. As shown, 
Family Rewards also increased the program group’s likelihood of paying off car and home 
loans, albeit these types of loans were incurred by small fractions of families participating in 
Family Rewards. 

Across the range of indicators examined here, it appears that Family Rewards contin- 
ued to demonstrate small but positive impacts on families’ financial security. At the time of 
the 42 -month survey, program group families were more likely to maintain bank accounts, 
have some savings, and be repaying debt. Unlike the early evaluation findings, the longer- 
term survey data do not show that the program had a positive effect on average household 
savings, but they do indicate that it continued to increase the program group’s likelihood of 
having “any” savings and savings over $500. Both groups reported similar levels of debt. 
However, by being more likely to pay down some types of debt (such as credit card bills), 
program group families were on a path to building and maintaining a credit history, a positive 
outcome for low-income families. Further, it is also possible that families diverted their extra 
income — and potential savings — to address immediate household needs, reduce hardships, 
and help improve their quality of life, issues that are examined in the next section. 


27 From the data, it is not possible to tell whether the balance is ongoing debt or whether it is a monthly 
balance to be paid off in the next credit payment cycle. 
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Material Hardship 

The early findings from the first two years of Family Rewards provide support for the pro- 
gram’s positive effects on material well-being. This section examines whether the positive 
effects observed in the short run were sustained — or even grew — in the early post-program 
period, when families could no longer boost their income with Family Rewards cash incentives. 

Unfortunately, a common definition of “material well-being” does not exist, nor is there 
a standard approach to its measurement. As a result, most efforts to measure this concept 
include scales or items that capture various dimensions of need — or hardships — across 
domains such as food adequacy, shelter, financial strain, and the like. The Family Rewards 42- 
month survey includes those types of measures to look more comprehensively at how families 
were faring at the time of the interview or in the recent past. Thus, depending on the measure, 
the outcomes reflect families’ circumstances during the program period or in the post-program 
period. The measures include: 

1. A multi-item material hardship index, based on responses to five commonly 
asked questions that assess whether families have experienced the following 
difficulties with housing or utilities in the past 12 months: not paid full rent 
or made a full mortgage payment, evicted for not paying rent or mortgage, 
unable to pay full utility bill, been without utilities, or had phone service dis- 
connected. 28 

2. A financial strain scale, with scores ranging from 4 to 16 — a lower score 
indicating greater perceptions of financial strain. The scale includes four 
statements, each capturing how strongly respondents feel about their finan- 
cial well-being (for example, “My financial situation is better than last year” 
and “I don’t worry about having enough money in the future”). 

3. An overall financial well-being measure that is assessed by asking respond- 
ents how their family finances usually work out at the end of the month, 
whether they have some money left over, just enough to make ends meet, or 
not enough money to make ends meet. 

4. A common self-reported measure of food insufficiency to assess the adequa- 
cy of food for the family in the past month. 29 


28 Material hardship scales were pioneered by Mayer and Jencks (1989). 

29 This same question has been used in United States Department of Agriculture surveys since the mid- 
1970s to measure food deprivation, and has been validated against other measures of hunger and nutritional 
adequacy; see Rose and Oliveira (1997). 
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5. Measures of whether families ever had to forgo medical care or medicine be- 
cause of costs in the past 12 months. 

The top panel of Table 3.4 focuses on hardships related to housing and utilities that 
were experienced over the 12 months preceding the 42 -month survey. For respondents, this 12- 
month period spans both Year 3 and the early post-program months, when families were 
beginning to experience the loss of additional income from reward payments. As shown, Family 
Rewards had no effect on the summary measure (“Any housing/utilities material hardships”). 
However, it did cause small but statistically significant reductions across a range of hardships in 
this domain, such as not paying full rent or mortgage payments or having utilities turned off. 

Assessing respondents’ financial circumstances more broadly, the survey’s financial 
strain scale asks respondents to compare their current financial situation with what it had been a 
year before. Exactly 51.4 percent of the program group reported that their current financial 
situation was better than it had been the year before, compared with 46.6 percent of the control 
group, a statistically significant increase of 4.8 percentage points. Program group members also 
scored a higher average “financial well-being” score (9.0 percent versus 8.8 percent), indicating 
that they were more likely to be positive about their financial well-being. Further, program 
group members were also less likely to report ever having borrowed cash from family and 
friends — 47.3 percent of the program group reported having done so, compared with 52.5 
percent of the control group, reflecting a statistically significant difference of 5.2 percentage 
points. Thus, across a range of measures, the program group’s perceived financial well-being 
appears to be more positive than that of the control group. 

Interestingly, when asked the same set of financial strain questions on the earlier 18- 
month survey, a much higher percentage (62.7 percent) of program group members rated their 
current financial situation better then than in the previous year, compared with 44.5 percent of 
the control group members who felt the same. Over time, reports of financial strain stayed 
relatively unchanged for control group respondents, but reports of perceived financial strain 
increased for the program group — in other words, by the 42 -month survey, fewer program 
group members than on the 1 8-month survey reported that their current financial situation was 
better than it had been in the last year. 

Focusing on the month before the interview, or on more recent experiences in the post- 
program period, results for an indicator of family finances also reveal positive effects. For 
example, the program group was more likely to report that they were able to “make ends meet” 
with the resources they had available (as shown under “Family finances usually work out...” 
in Table 3.4). Roughly 35 percent of the program group reported that they did not have enough 
to make ends meet at the end of the month, compared with 4 1 percent of the control group, a 
statistically significant reduction of 5.6 percentage points on this measure of hardship. 
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Table 3.4 

Impacts on Material Hardship and Financial Strain 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Effect 

Size 

Any housing/utilities material hardship in 

the past 12 months (%) 

55.3 

58.8 

-3.5 

0.118 


Did not pay full rent or mortgage 3 

40.0 

44.1 

-4.2 * 

0.061 


Evicted from home for not paying 
rent or mortgage 3 

3.9 

4.5 

-0.5 

0.554 


Did not pay full utility bill b 

30.7 

32.3 

-1.6 

0.444 


Utility was turned off b 

5.5 

8.0 

-2.5 ** 

0.028 


Phone service was disconnected 0 

19.8 

22.3 

-2.5 

0.169 


Financial well-being (4 = low; 16 = high) d 

9.0 

8.8 

0.3 ** 

0.017 

0.108 

Strongly or somewhat agree with 
the following (%) 

Financial situation is better than last year 

51.4 

46.6 

4.8 ** 

0.034 


Do not worry about having enough 
money in future 

20.5 

20.2 

0.3 

0.852 


Can generally afford to buy needed things 

67.8 

65.4 

2.4 

0.258 


Sometimes have enough money to buy 
something or go somewhere just for fun 

30.3 

28.0 

2.3 

0.272 


Family finances usually work out to have 
the following at end of month (%) 

Some money left over 

14.9 

12.5 

2.4 

0.123 


Just enough to make ends meet 

49.7 

46.5 

3.2 

0.152 


Not enough to make ends meet 

35.4 

41.0 

-5.6 *** 

0.009 


Ever borrow cash from family or friends (%) 

47.3 

52.5 

-5.2 ** 

0.021 


Ever sell personal belongings 

at a pawnshop (%) 

14.6 

18.3 

_3 7 ** 

0.027 


Children skipped meal in prior month (%) 

3.6 

6.5 

_2 9 *** 

0.004 


Food security (1 = low; 4 = high) 0 

3.4 

3.3 

q y *** 

0.001 

0.144 

Insufficient food 1 (%) 

15.3 

20.7 

_5 4 *** 

0.002 


Did not get needed medical care because 

of cost in past 12 months* (%) 

7.1 

8.1 

-l.i 

0.371 


Did not fill prescription because 
of cost in past 12 months (%) 

14.4 

13.0 

1.3 

0.393 


Sample size (total = 1,982) 

1,024 

958 





(continued) 
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Table 3.4 (continued) 

SOURCE: MDRC calculations using data from the Family Rewards 42-month survey. 

NOTES: Sample sizes may vary because of missing values. 

The items in this section of the survey were administered to a random subsample (N = 1,982) of 
the survey respondents. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the difference between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 

5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

The effect size is the difference between program and control group outcomes expressed as a 
proportion of the standard deviation of the outcome for both groups combined. 

“Only about 4 percent of the survey sample (N = 130) owned an apartment or a house at the time 
of the survey. 

•“Utilities include gas, oil, and electricity. 

“This includes cellular or land service. 

d Components of the financial well-being scale have been coded such that a lower score implies 
being worse off and a higher score implies being better off. The scale is calculated by summing 
responses to the four component questions. Thus, the financial well-being scale presented here 
ranges from 4 to 16 points. 

“The food security question describes food eaten by the family in the prior month: 1= Often not 
enough to eat; 2 = Sometimes not enough to eat; 3 = Enough to eat but not always the kinds of food 
desired; 4 = Enough to eat of the kinds of food desired. 

Insufficient food is defined as "sometimes" or "often times" not having enough food to eat. 

g This excludes prescriptions. 

The program also reduced reports of food insufficiency, which is measured with a ques- 
tion that asks respondents to describe the food that they and their families have eaten in the prior 
month: (1) often not enough to eat, (2) sometimes not enough to eat, (3) enough to eat, but not 
always the kinds of food desired, and (4) enough to eat of the kinds of food desired. Overall 
scores close to 1 indicate that families often do not have enough to eat. Values close to 4 
indicate that families have enough of the kinds of food desired. Households reporting that they 
sometimes or often do not get enough to eat are termed “food-insufficient.” The results suggest 
that program group members were less likely than the control group members to report food 
insufficiency — in other words, that they sometimes or often did not get enough to eat (15.3 
percent versus 20.7 percent, respectively, for the program and control groups, a statistically 
significant drop of 5.4 percentage points). 

Also reflecting hardship, the survey asked respondents whether they were unable to get 
needed medical care because of costs in the prior year, and whether they were unable to fill 
prescriptions for the same reason. Eight percent to 13 percent of control group members 
indicated that they had incurred these hardships, and the rates were comparable for the program 
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group participants. As described later in this report, health coverage rates were very high for the 
study sample, which may have ensured their access to needed medical care. 

Overall, consistent with the early evaluation results, the pattern of findings across 
the range of measures in this section provides evidence that Family Rewards, even after the 
payments ended, continued to improve families’ financial security and material well-being 
— particularly their perceived financial well-being and reported food security. (In both 
cases, it appears that reports of hardships are increasing for control group members but are 
staying about the same for program participants.) However, recognizing that the material 
well-being experiences for families who were interviewed closer to the end of the program 
may be more positive than for those who were interviewed many months after they stopped 
receiving the cash transfers, an additional analysis was conducted to look at their well-being 
impacts separately. Consistent with expectations, as shown in the supplement to this 
report, 30 the program’s positive effects on material well-being and hardship appear to be 
concentrated among the families who were interviewed soon after they had completed their 
three years in the program — in other words, they were eligible to earn rewards for a 
significant portion (seven months or more) of the prior 12-month period that is used here to 
examine various aspects of material well-being. 

Reports of growing hardships, in the early months after the program ended, were cap- 
tured in the interviews that were conducted with participating parents and their children. In one 
case, a high school student described how her mother was coping to cover household expenses: 

Now [that the program is over], my mom needs more help, like, for buying gro- 
ceries and for money for laundiy, and if we need some little things in the house 
that we’re missing she’s always asking, “Do you guys have extra money that you 
can lend me?” And I know she asks her siblings. Her own siblings, but some- 
times she has to ask our neighbor for money. ... I know before the program she 
had to... do that sometimes. And... while we were in the program it was, like, 
less frequent and now it’s become more frequent. 


Housing Stability 

In the early stages of the evaluation, some participants in the in-depth interviews discussed the 
strides they had made in improving their housing circumstances. One participant reported using 
the payments to move to a bigger apartment, and another was able to pay off debt that was 
preventing her from getting credit to purchase a house. The 42 -month survey provides longer- 
term evidence on the topics of housing status and residential stability (that is, whether people 
stay or move from the neighborhood in which they were living at the time of random assign- 

30 See supplementary Appendix Table F.9 in Riccio et al. (2013), available at www.mdrc.org. 
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ment). The Family Rewards sample members, who all lived in high-poverty neighborhoods at 
study entry, may be vulnerable to housing instability, which, in turn, may have consequences 
for child and family well-being. Some studies report an association between high mobility and 
poor school outcomes for children in low-income families. 31 

Family Rewards survey respondents were asked to report on their housing status — 
whether they owned a home or rented, or received some form of housing assistance (Section 8, 
public housing, or other form of housing subsidy). They were also asked whether they had 
moved since random assignment and, if they had, to describe the reasons they moved. Residen- 
tial mobility is high among low-income populations, and families move for a variety of reasons, 
including a desire for safety, a better neighborhood, a bigger place, and proximity to good 
schools and jobs, or because they are unable to pay the rent at their current residence. 

Table 3.5 shows that Family Rewards did not have an effect on housing status or resi- 
dential mobility. Both the program and control group respondents were equally likely to have 
moved after random assignment (21.6 percent versus 21.4 percent) and they offered generally 
the same types of reasons for moving. 32 


Marriage and Family Composition 

The data in Table 3.5 suggest that Family Rewards may have had small effects on family 
composition, including marital status. According to the 42-month survey, roughly equal 
percentages of respondents were married and living with a spouse in the two groups (18.7 
percent versus 17.8 percent in the program and control groups, respectively). However, program 
group members were somewhat more likely to report that they were divorced (15.4 percent 
versus 13.1 percent, a statistically significant increase of 2.3 percentage points). What might 
explain this effect? It was first documented in the early evaluation report and appears to persist 
over time. 33 Although only speculative, it is possible that the increased financial stability that 


3 'Simpson and Fowler (1994); Crowley (2003); Burkam, Lee, and Dwyer (2009). 

32 Nationally, roughly 12 percent of the U.S population moved to a new address in 2008; see the Bureau of 
Labor Statistics Web site at www.bls.gov. However, mobility rates are higher among the low-income popula- 
tions, renters, and younger populations, but these rates can vary according to local housing markets, which can 
limit mobility options; New York City is one example of a tight housing market with limited housing opportu- 
nities. One recent study of a cross-section of residents in 10 low-income neighborhoods found that roughly 57 
percent of the sample had moved from their original housing unit in a three-year survey period; see Coulton, 
Theodos, and Turner (2009). 

33 Similarly, the early evaluation (Riccio et al., 2010) found some evidence suggesting that Family Re- 
wards may have had small effects on marital status, increasing to a small degree the likelihood of marriage. 
The follow-up survey does not provide support for that finding. 
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The Opportunity NYC Demonstration: Family Rewards 

Table 3.5 


Impacts on Housing and Family Composition 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Housing status and residential mobility (%) 

Current housing arrangement 

Owns home or apartment" 1 

4.8 

4.0 

0.9 

0.237 

Rents home or apartment" 1 

89.5 

89.9 

-0.4 

0.708 

Lives with family or friends 

Contributes to rent 

3.6 

3.7 

-0.1 

0.854 

Does not contribute to rent 

0.9 

1.0 

-0.2 

0.674 

Other 

1.2 

1.4 

-0.2 

0.687 

Currently lives in public or subsidized housing 

67.4 

68.0 

-0.6 

0.714 

Currently lives in New York City 

97.6 

98.1 

-0.6 

0.281 

Moved since random assignment 

21.6 

21.4 

0.2 

0.902 

Family composition (%) 

Current marital status 

Single, never married 

45.9 

47.3 

-1.5 

0.364 

Married and living with spouse 

18.7 

17.8 

0.9 

0.442 

Separated or living apart from spouse 

16.0 

17.1 

-1.2 

0.395 

Divorced 

15.4 

13.1 

2.3 * 

0.067 

Widowed 

4.2 

4.7 

-0.6 

0.433 

Living with partner 

10.5 

9.5 

0.9 

0.406 

Number of children* 5 

0 

2.2 

2.2 

0.0 

0.964 

1 

30.4 

33.2 

-2.8 * 

0.088 

2 

32.6 

32.7 

-0.1 

0.959 

3 or more 

34.8 

31.9 

2.9 * 

0.082 

Had or fathered a child since random assignment 

8.8 

7.6 

1.2 

0.218 

Had or fathered a child out of wedlock 

6.0 

5.0 

1.1 

0.190 

Sample size (total = 2,966) 

1,543 

1,423 




SOURCE: MDRC calculations using data from the Family Rewards 42-month survey. 


NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the difference between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 
5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

a Only about 4 percent of the survey sample (N = 130) owned an apartment or a house at the time 
of the survey. 

b This measure only includes children 18 years of age or younger. 
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some participants experienced over the course of their participation in the program may have 
led them to feel better positioned to divorce. It should be underscored that the overall effects 
on divorce are consistent and small, and the mechanisms through which the program might 
have influenced those results are not understood. 

Table 3.5 also shows that Family Rewards had a small effect on family composition, 
with more program group members than control group members reporting three or more 
children living with them. Since the program did not have an effect on childbearing, it is 
likely that the effect on the number of children in the household is driven by the dynamics of 
children moving in and out of the household — that is, changes in their living arrangements. 


Impacts for Key Subgroups 

Because overall impacts can mask program effects for some groups of study participants, the 
income and material well-being impacts of Family Rewards are analyzed for subgroups defined 
by baseline measures of education, employment, and income. The subgroups were prioritized 
based on the understanding that indicators of well-being vary greatly across groups defined by 
socioeconomic characteristics, the possibility that groups defined by those characteristics might 
have differential responses to the program’s cash incentives, and that different capacities or 
starting points might affect how individuals respond to the incentives. For example, the more 
“advantaged” participants — those with higher income or those who were employed or who 
had higher levels of education at study entry — may be more likely to earn rewards and thus 
experience greater improvements in income, poverty, and material well-being. Descriptive 
analyses discussed in Chapter 2 of this report also reveal significant variations across the top 
and bottom 20 percent of reward earners. Among other distinguishing characteristics, the top- 
earning families were more likely to have been employed at baseline and to hold a high school 
diploma or a General Educational Development (GED) certificate. 

To examine program impacts for subgroups with different starting advantages or disad- 
vantages, the effects of Family Rewards were analyzed separately for parents who did not have 
a high school diploma or GED certificate at the time of random assignment, and for those with 
at least a high school diploma or GED certificate. A similar approach was used to examine the 
program impacts for the subgroups who were defined by employment and income at random 
assignment. In general, impacts are expected to vary to some extent across subgroups, simply as 
a result of natural variation around the average impact for the full sample. This section exam- 
ines whether that variation in impacts across subgroups is statistically significant, or beyond 
what would be expected to occur naturally. For that reason, the focus is not on whether a given 
impact for, say, the less educated subgroup is statistically significant, but whether the difference 
between that impact and the impact for the more educated subgroup is statistically significant 
(which is indicated by daggers in the rightmost column of the tables). If the difference between 
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these two impacts is not statistically significant, the results suggest that the effects observed for 
the full sample generally hold across both groups compared. 

About 40 percent of the parents in the study sample did not have a high school diploma 
or a GED certificate when they enrolled in Family Rewards. The program group that is defined 
by a high school diploma or GED certificate at baseline earned an average $9,563 in cash 
rewards over the three-year period, a difference of $2,175, or 29 percent, compared with the 
average reward earnings of $7,388 for those without a high school diploma or GED certificate. 
While both groups supplemented their household income with substantial cash rewards, one 
question is whether the differential in the amounts of rewards earned translates into different 
quality-of-life and well-being outcomes for the two groups. Findings that are presented in the 
supplement to this report suggest that the program impacts on income and financial well-being 
were largely comparable for the groups with and without a high school diploma or GED 
certificate. 34 On the measures that were examined, the small differences in impacts across the 
subgroups were not statistically significant. 

A similar analysis was conducted comparing subgroups that are defined by parents’ 
employment status at the time of random assignment. Fifty-three percent of the parents who 
enrolled in Family Rewards were working at the time of study entry. Over the course of the 
three years, the group working at baseline earned an average $10,095 in cash rewards, com- 
pared with their nonworking program group counterparts, who earned closer to $7,203, a 
difference of $2,892, or 40 percent. Despite these differences in total rewards earned, results 
shown in the supplement to this report suggest that Family Rewards had similar effects on the 
income and financial well-being measures whether or not the parent was working upon entry 
into the study. 35 

The last subgroup examined here, in Table 3.6, is distinguished by parents’ income sta- 
tus at random assignment — that is, parents with income at or above 50 percent of the federal 
poverty level and those with income below 50 percent of the federal poverty level. Baseline 
income both captures the families’ level of economic disadvantage at enrollment in the study 
and the relative value of the rewards for them — for instance, a family with two children in the 
program could earn up to $3,000 from the children’s education rewards, more than $2,000 from 
health rewards, and more than $2,000 from the parents’ educational and training rewards. This 
potential $7,000 would represent a bigger relative share of income for a family starting out in 
severe poverty relative to those starting out less poor. Thus, the “incentive value” might be 
greater for those in severe poverty, and it may have a bigger relative consequence for their 
living conditions. 


34 See supplementary Appendix Table F.7 in Riccio et al. (2013), which is available at www.mdrc.org. 
35 See supplementary Appendix Table F.8 in Riccio et al. (2013), which is available at www.mdrc.org. 



The Opportunity NYC Demonstration: Family Rewards 

Table 3.6 

Impacts on Income, Poverty, and Material Hardship, 
by Respondent’s Poverty Level at the Time of Random Assignment 



Program 

Control 

Difference 



Subgroup and Outcome 

Group 

Group 

(Impact) 

P- Value 

Sig. 

Income at or above 50% of FPL 
at baseline 

Household income at or below the federal 






poverty level (including Family Rewards 
payments) a,b (%) 

51.1 

62.1 

-110 *** 

0.000 


Household income at or below the federal 
poverty level (excluding rewards) c (%) 

62.4 

62.1 

0.2 

0.903 


Average total household income in month 
prior to interview (including rewards) a,b ($) 

2,093 

1,771 

323 *** 

0.000 


Insufficient food d (%) 

16.0 

19.0 

-3.0 

0.181 

t 

Did not pay full rent or mortgage in 
past 12 months 6 (%) 

42.9 

42.0 

0.9 

0.764 

tt 

Usually not enough money to make ends 
meet at end of month (%) 

34.7 

38.1 

-3.4 

0.227 


Average savings more than $500 f (%) 

13.7 

11.7 

2.0 

0.304 


Sample size (total = 1,193) 

642 

551 




Income less than 50% of FPL 






at baseline 






Household income at or below the federal 






poverty level (including Family Rewards 
payments) a,b (%) 

63.5 

77.2 

-13.6 *** 

0.000 


Household income at or below the federal 
poverty level (excluding rewards ) 6 (%) 

71.9 

77.2 

-5.3 * 

0.088 


Average total household income in month 
prior to interview (including rewards) a,b ($) 

1,781 

1,409 

372 *** 

0.000 


Insufficient food d (%) 

14.0 

23.2 

_g 2 *** 

0.001 

t 

Did not pay full rent or mortgage in 
past 12 months 6 (%) 

35.7 

46.4 

- 10.8 *** 

0.002 

tf 

Usually not enough money to make ends 
meet at end of month (%) 

36.4 

45.2 

- 8.8 ** 

0.012 


Average savings more than $500 f (%) 

10.3 

6.0 

4.2 ** 

0.035 


Sample size (total = 788) 

382 

406 





(continued) 
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Table 3.6 (continued) 

SOURCE: MDRC calculations using data from the Family Rewards 42-month survey. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the differences between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical significance 
levels (Sig.) are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

FPL = federal poverty level. 

a Family Rewards payments are based on Seedco's Family Rewards data from program Year 3, 
which include activities completed in September 2009 through August 2010. The monthly Family 
Rewards payment amount is calculated by dividing the annual reward amount by 12. The payment data 
do not include bonus payments that some families received for opening new bank accounts. 

b Annual household income is calculated by multiplying by 12 the respondent's income in the month 
prior to the survey interview. For program group members, it includes Family Rewards payments earned 
during program Year 3; see the preceding note. The federal poverty level was calculated based on 
annual income (monthly income multiplied by 12) and the household size at the time of the survey. The 
poverty threshold was measured according to the 2010 or 201 1 Poverty Guidelines, depending on when 
a respondent was interviewed. 

c Annual household income is calculated by multiplying by 12 the respondent's income in the month 
prior to the survey interview. This calculation excludes the Family Rewards payments earned during 
program Year 3. The federal poverty level was calculated based on annual income (monthly income 
multiplied by 12) and the household size at the time of the survey. The poverty threshold was measured 
according to the 2010 or 201 1 Poverty Guidelines, depending on when a respondent was interviewed. 

insufficient food is defined as "sometimes" or "often times" not having enough food to eat. 

c Only about 4 percent of the survey sample (N = 130) owned an apartment or a house at the time of 
the survey. 

f About 6 percent of the sample is excluded from this analysis because of missing data. 


Over the three-year program period, the households in severe poverty (that is, below 50 
percent of the federal poverty level) at random assignment earned, on average, about $7,368 in 
reward payments, compared with the $9,469 earned by participants in less severe poverty (not 
shown in table). The difference in cash rewards earned appears to be largely driven by the 
inability of the very poor to earn the workforce rewards. Both groups appear to have relatively 
similar earnings for rewards in two of the three domains — education and health. Among those 
with household income under 50 percent of the federal poverty level, as shown in Table 3.6, 
Family Rewards reduced the proportion of families reporting food insufficiency by about 9 
percentage points, compared with the 3 percentage point reduction for their higher-income 
counterparts. It also reduced the proportion of households below 50 percent of the federal 
poverty level who were not paying their full rent or mortgage (by -10.8 percent versus 0.9 
percent for the higher-income group, a difference in impacts that is statistically significant). Even 
though the very low-income families earned less in reward payments than their higher-income 
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counterparts, the data seem to suggest that the additional income from Family Rewards may have 
made a bigger difference in their lives and had a greater impact on their material well-being — 
the rewards accounted for 12.7 percent of Year 3 income for the families in severe poverty, 
compared with 14.5 percent for those with income at or above the poverty level (not shown). 

Based on the subgroups that are examined here, the program does not appear to have had 
different effects on material well-being depending on parents’ readiness for work (that is, their 
education or work status at enrollment). There is some evidence, however, that families with 
very low income at enrollment experienced more positive improvements on some indicators of 
well-being than their higher-income counterparts. In addition to the subgroups that are discussed 
here, variation in income and material well-being impacts was examined for other subgroups, 
including those defined by household composition, housing status, and food stamp receipt. These 
analyses, included in the supplement to this report, 36 point to few noteworthy effects. 


Conclusion 

Family Rewards was launched as a bold new intervention to address intergenerational poverty 
among low-income families. Achieving that goal meant that families had to take the necessary 
steps to earn rewards that were conditioned on a variety of activities designed to improve their 
human capital and overall well-being. 

Drawing extensively on the 42 -month survey, this chapter explored the extent to which 
program participation — and the cash rewards earned — reduced household poverty and 
improved the economic and material well-being of program participants. The results presented 
here provide evidence that supports Family Rewards’ continued effectiveness in reducing short- 
term poverty and income-related hardships while the program was in effect: as long as families 
were eligible to earn rewards, and they were able to supplement their household income with 
cash transfers, the program produced significant income gains for participating households. It 
reduced the proportion of families who were living in poverty, including severe poverty. It 
reduced the proportion of families who were suffering from food insufficiency or financial 
strain, or reporting different types of housing-related material hardships. However, these effects 
were mostly driven by the cash transfers themselves, and there is little evidence that the pro- 
gram continued to have these effects long after they ended. 

The report turns next to an examination of the effects of Family Rewards on program par- 
ticipants’ human capital development. Chapter 4 focuses on Family Rewards’ effects on chil- 
dren’s education; Chapter 5 examines the program’s impacts on family health care practices and 
outcomes; and Chapter 6 explores the program’s effects on parents’ employment and training. 

36 The supplement to this report is available at www.mdrc.org; see Riccio et al. (2013). 
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Chapter 4 

Effects on School Progress and Performance 


Family Rewards sought to break the cycle of intergenerational poverty by increasing children’s 
school perfonnance and narrowing the achievement gap between children in low-income families 
and their higher-income counterparts. 1 The program provided a range of incentives tied to 
school-related behaviors, including incentives for attendance, parental engagement, student 
performance on tests, and other indicators of achiev ement. 

As shown in the earlier MDRC report on Family Rewards, after two years the program 
had mixed effects on students’ school perfonnance. 2 Although Family Rewards did lead to small 
increases in engagement among parents of elementary and middle school students, it had no 
effects on these students’ school progress. However, the program did lead to notable and 
encouraging effects for ninth-graders who were more academically prepared than their peers, 
increasing their attendance, the number of credits they earned, and the number of Regents 
exams they had passed by the end of Year 2. 

This chapter updates the findings through Year 4. It examines whether effects emerged 
for the younger students (in elementary and middle school) as they aged, and whether the 
encouraging effects for the more academically prepared ninth-graders persisted and led to 
higher graduation rates. Finally, it examines whether the effects persisted beyond the third year, 
when the rewards were no longer offered. 

Although the program did not affect the school progress of younger students during the 
first two years, as noted in the earlier report, it is plausible that effects could emerge during or 
after Year 3. First, on average, the school perfonnance of children from low-income families 
declines as they age, with many children falling further and further behind their higher-income 
peers. 3 The incentives may have helped stem the extent to which these children become disen- 
gaged from school. Second, Family Rewards transferred substantial sums of money to partici- 
pating families, reducing poverty and material hardship. Other research suggests that these 
increases in family income and well-being can affect school perfonnance by themselves. 4 


'See Lee and Burkham (2002) for further evidence on this achievement gap. 

2 Riccio et al. (2010). 

3 Carneiro and Heckman (2003). 

4 Dahl and Lochner (2012); Morris and Gennetian (2003). In addition, positive effects in the other do- 
mains, such as improved health, can also lead to better school outcomes; see Romero and Lee (2008). See, 
also, the Family Rewards logic model in Chapter 1, Figure 1.1. 
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Overall, the findings through Year 4 continue the story from Year 2. The program had 
no effect on the school performance of elementary and middle school students, but continued to 
increase the performance of proficient ninth-graders, or those ninth-graders who scored at the 
proficient level or higher on their eighth-grade standardized tests. The latter group showed large 
increases in attendance and credits earned during the first three years. In addition, the effects 
lasted into the fourth year for ninth-graders who were proficient on the English language arts 
(ELA) test, although not for those who were proficient on the math test. By the end of the fourth 
year, ninth-graders who entered the study proficient on the ELA test were more likely than their 
control group counterparts to have passed the required number of Regents exams necessary for 
a diploma, and they were more likely to have graduated — by 8 percentage points. The effects 
for this group are encouraging and on a par with those found from other more intensive, school- 
based interventions. 5 

The Education Rewards Offer 

The education incentives were intended to encourage both achievement and the effort that 
supports achievement. For this reason, the program rewarded a variety of behaviors, listed in 
Table 4.1. Key differences between the rewards by age are that high school students can earn 
considerably more than can younger students (given that attendance and performance tend to 
decline with age) and that some or all of the rewards they earn are paid directly to them, rather 
than to their parents (given that high school students have more direct control over their educa- 
tional effort than do younger students). 

By rewarding a variety of activities, the program ensures that most families will receive 
at least some payments, even if they are not able to meet every benchmark. At the same time, 
the program can achieve its goal of immediate poverty reduction by transferring significant 
resources to low-income families. For example, a single parent with one child in middle school 
and one child in high school could earn more than $3,000 per year through the education 
rewards alone if she and her children met all or most of the benchmarks. 

As noted in Table 4.1 and discussed in more detail in Chapter 2, several rewards 
were discontinued after the second program year, based on cost considerations and the early 
impact findings. For example, the attendance reward was discontinued for elementary and 
middle school students. Average attendance rates were already fairly high for students in 
those grades, as shown by the control group, and the program produced little effect on this 
outcome through Year 2. 


5 Quint, Bloom, Black, and Stephens (2005); Kemple, Herlihy, and Smith (2005); Bloom and Unterman 
( 2012 ). 
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The Opportunity NYC Demonstration: Family Rewards 

Table 4.1 

Education Rewards 

Activity Payment 


Attendance: 95% or higher 
Discontinued for Year 3 for elementary and 
middle school students 

English language arts (ELA) and math 
standardized tests: scoring at proficiency level 
or improving 1 level 

Regents exams: passing 

Credits: earning 1 1 or more per year 
PSAT : taking the test 

High school graduation 

Having a library card 
Discontinued for Year 3 

Attending parent-teacher conferences 

Parent meetings with teachers to discuss 
annual ELA and math test results 

Discontinued for Year 3 


$25 per month for elementary/middle school 
students; $50 per month for high school students' 1 

$300 for each test for elementary school students; 
$350 for each test for middle school students 

$600 for each test for high school students, up to 5 
tests' 5 

$600 for high school students per year a 

$50 per test, for taking the test up to 2 times 
(maximum of $ 1 00 per student) 15 

$400 once a 

$50 once during the program (all grades) 0 

$25 per conference, twice per year 

$25 once per year for elementary/middle school 
students 


NOTES: “Half of the payment was made directly to the student, and half was made to the parent. 
b The entire payment was made directly to students who were in high school. 
c The entire payment was made directly to students who were in high school; for elementary and 
middle school students, payment was made to the parent. 


Data and Samples 

This chapter uses data from several sources. First, administrative payment data from the 
program provide information about the receipt of rewards during all three years of program 
operation. These records provide data on the number and type of rewards earned, as well as the 
total amount earned by families and students. 
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Data on key education outcomes are obtained from administrative records provided by 
the New York City Department of Education (DOE). 6 These data are available for all students 
in the study for one year before study entry, or school year 2006-2007, and for four years after 
study entry, or through school year 2010-2011. School outcomes that are available from the 
DOE records include attendance rates, scores on annual math and English language arts (ELA) 
tests, performance on Regents exams, course credits earned, and school enrollment status. 
Although these data do not provide information for students attending parochial schools, private 
schools, or schools outside New York City, survey data shown later indicate that few students in 
the sample attended these other types of schools. 

Finally, a survey was administered about 42 months after study entry to a random subset 
of parents. The survey provides infonnation on intennediate outcomes, such as parental effort 
(parents’ interaction with their children and their children’s teachers), children’s engagement in 
extracurricular activities, and parents’ ratings of their children’s school performance. See 
Appendix J, in the supplement to this report, for a survey description and a response analysis. 7 

Although families who were targeted for the study had to have at least one child in the 
fourth, seventh, or ninth grade, once they were enrolled in the program all their school-age 
children were eligible for rewards. 8 Over 9,000 children were school-age when they enrolled in 
the study, and nearly 60 percent of these children were entering one of the three target grades. 
The analysis of education outcomes presented in this chapter focuses on students in each of the 
three target grades, meaning students who were set to enter fourth, seventh, or ninth grade when 
they enrolled in the study. The focus is on a specific grade cohort, rather than a broader cohort 
of “all elementary-age students,” for example, for two reasons. First, the findings for a one- 
grade cohort are more easily interpreted than those for a broader group. It is easier to assess how 
a cohort is faring when all students in that cohort started out in the same grade — for example, 
all entering ninth-graders should be in twelfth grade by Year 4, having earned 44 or more 
credits and passed five or more Regents exams. A cohort that is made up of ninth- through 
twelfth-graders, in contrast, will be at different points by Year 4. The second reason for focusing 
on students in the target grade only is that they make up the majority of the sample. Seventh- 


6 Students in the target grades (fourth-, seventh-, and ninth-graders) were matched to the DOE data using 
their DOE student ID, which was known to the study team, given that these youth were identified as eligible 
for the study using DOE records. However, parents were not required to provide a student ID when enrolling 
other children (siblings of the target children) into the study, since few would have known it. For those children, 
student IDs were obtained by comparing name, date of birth, and other information on the study enrollment 
form with similar data on the DOE records. Using this matching process, MDRC was able to obtain student 
IDs for more than 90 percent of students in the Family Rewards study. 

7 The supplement to this report is available at www.mdrc.org; see Riccio et al. (2013). 

8 These three grades were chosen because they represent key transition periods in a child’s school trajecto- 
ry. See Riccio et al. (2010) for more detail. 
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graders, for example, make up 65 percent of the middle school group. Results for the full 
sample in each age range (elementary school, middle school, and high school) are presented in 
the supplement to this report. 9 The results are similar to those presented here. 

All impacts are estimated using an ordinary least squares regression model that controls 
for a range of background characteristics, such as the student’s race/ethnicity, gender, and test 
scores from the preceding year, and parents’ education level, marital status, and employment 
status. 10 The standard errors of the impact estimates are adjusted to account for the potential 
clustering of student outcomes at the family level. Finally, as noted in earlier chapters, impacts 
are examined for multiple outcomes within each area — parental engagement, student attend- 
ance, student performance, and so on. Given that the likelihood of finding “false positives” 
increases as the number of outcomes increases, caution should be used when interpreting 
impacts that do not appear to be part of a larger pattern of impacts within a given area. 


School Progress Through Year 4 for the Control Group 

Students who enrolled in the study came from low-income families in six of New York’s 
highest-poverty neighborhoods. As such, they face a range of challenges to their school pro- 
gress. Children from low-income families in general can be distinguished from those in higher- 
income families as early as kindergarten, exhibiting lower scores on a range of achievement and 
school readiness measures. 11 These early disparities grow over time. 12 By high school, for 
example, low-income students are nearly three times more likely than higher-income students to 
have repeated a grade and nearly six times more likely to have dropped out. 13 Data for students 
in the control group for this study bear this out. 

Figure 4. 1 presents school progress over the four years following study enrollment for 
control group students in the three target grades. 14 Students who were “on grade” were in the 
grade they should be in if they had been promoted each year. Consider fourth-graders first. By 


9 See supplementary Appendix Tables G.l through G.3 in Riccio et al. (2013), at www.mdrc.org. 

10 Specifically, the regression model includes the following variables: community district, average class size 
for grade and school attending at study entry, prior year’s math score, prior year’s ELA score, male, African- 
American, special education status, number of children in the household, English is the primary language spoken 
at home, two-parent family, mother’s education level, mother’s employment status, and presence of mother in 
the home. 

n Lee and Burkam (2002). 

12 Cameiro and Heckman (2003). 

13 Planty et al. (2009). 

14 The data underlying these figures are shown in supplementary Appendix Table G.4 in Riccio et al. 
(2013), which is available at www.mdrc.org. 
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The Opportunity NYC Demonstration: Family Rewards 
Figure 4.1 

School Progress During the Four- Year Study Period, Control Group 
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tive records. 

a 

NOTES: The math score is divided by 10 for ease of presentation. For example, a score of 610 is 
shown on the graph as 61. 

b 

The follow-up year refers to the expected grade, assuming yearly promotion. 


Year 4, only 86 percent of the children in the control group who began the study in fourth grade 
had progressed to seventh grade. About one-third of the remaining 14 percent of the control 
group students had been held back at some point, while the remainder were no longer attending 
a New York City public school. The figure also shows scores and proficiency ratings for the 
standardized math tests administered to all New York State students in the third through eighth 
grades. In Year 1, for example, over 70 percent of control group students who were entering the 
fourth grade scored at the proficient level or higher on the math test, with an average score of 
669 (depicted in 10s in Figure 4.1). While average scores stayed fairly constant over the four 
years, proficiency rates dropped to just over 40 percent in Years 3 and 4. 

In 2010, amid concerns that these tests had become too easy and not reflective of the 
proficiency needed in each grade, the New York State Department of Education raised the 
scores necessary to be deemed proficient. In 2009 and earlier, a score of 650 or higher was 
required in order to be deemed proficient. Starting in 2010, the cutoff scores were raised to 658 
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to 684, depending on the grade level and the test. As a result, proficiency rates citywide fell 
dramatically between 2009 and 2010, as they did for students in the Family Rewards study. 15 

The second panel of Figure 4.1 presents data for entering seventh-graders in the control 
group. This group shows a steeper decline in on-grade status, particularly between Years 3 and 
4, when most of them should have moved from ninth to tenth grade. By Year 4, only about 74 
percent of these students were enrolled in tenth grade. And while the majority were enrolled in 
tenth grade in Year 4, only about half of these students (45.7 percent) had accumulated 22 
credits (that is, were on grade for credits earned) by the end of that year. 

The final panel presents data for entering ninth-graders. The percentage of these stu- 
dents who were on grade plummeted over the years, particularly between the ninth and tenth 
grades (Years 1 and 2 of the study). By Year 4, only 51 percent of this group was enrolled in 
twelfth grade. About 28 percent of these students were enrolled in a lower grade, and most of 
the remaining 2 1 percent were listed on the DOE records as having dropped out or transferred 
out of the New York City school system at some point during the four-year study period. 
Attendance was also lower in the higher grades — among those enrolled in Year 4, for example, 
the average attendance rate was 76 percent. Finally, only 48 percent of these students had graduat- 
ed by the end of Year 4, or one in two students who had started ninth grade four years earlier. 16 


Receipt of Education Rewards 

The previous report on Family Rewards illustrated that, through Year 2, most students in the 
program had earned at least one education reward in each year, with rewards for attendance 
being the most common. 17 Table 4.2 presents reward receipt over the full three-year program 
period. 18 (Chapter 2 presents a discussion of reward receipt and families’ engagement with the 
program more generally over the three-year period.) 

For entering fourth-graders, reward receipt fell substantially from Year 2 to Year 3. Only 
66.8 percent of these students earned a reward in Year 3 (compared with 94.2 percent in Year 2), 


15 Medina (2010). 

16 The graduation rate (48 percent) is somewhat higher than the percentage of students who had accumulat- 
ed 44 credits or more through Year 4 (41 percent). This discrepancy appears to occur in part because a few 
high schools do not use the standard system of reporting credits, which leads to an underreporting of credits 
earned toOE. In addition, students with Individualized Education Plans (lEPs) — mandated by the Individuals 
with Disabilities Education Act, to help children with disabilities achieve their educational goals — can earn an 
IEP diploma without earning 44 credits or more. This diploma, which is not recognized by New York State as 
a regular diploma, will no longer be offered starting in 2013. 

17 Riccio et al. (2010). 

ls Reward receipt for the full sample of children in each age range is presented in supplementary Appendix 
Table G.5 in Riccio et al. (2013), which is available at www.mdrc.org. 
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The Opportunity NYC Demonstration: Family Rewards 

Table 4.2 

Education Rewards Earned, by Grade Level at the Time of Random Assignment 






Years 1, 2, and 3 

School Level and Outcome 

Year 1 

Year 2 

Year 3 

Combined 

Students entering grade 4 





Earned any education reward (%) 

97.5 

94.2 

66.8 

99.0 

Total amount earned 1 ($) 

604 

651 

342 

1,446 

Earned at least 1 attendance reward (%) 

87.0 

76.5 

— 

91.8 

Earned an attendance reward in more than 4 periods (%) 

46.9 

38.9 

- 


Earned reward for English language arts (ELA) test b 

49.8 

71.8 

26.2 

83.1 

Earned reward for math test b 

66.6 

78.1 

38.5 

88.8 

Earned reward for obtaining a library card (%) 

66.2 

7.9 

- 

74.1 

Parent earned a reward for attending parent-teacher 
conference (%) 

67.8 

63.5 

47.9 

80.9 

Sample size 




862 

Students entering grade 7 





Earned any education reward (%) 

97.6 

89.2 

82.4 

99.0 

Total amount earned 11 ($) 

637 

585 

1,203 

2,156 

Earned at least 1 attendance reward (%) 

84.2 

71.8 

67.6 

91.5 

Earned an attendance reward in more than 4 periods (%) 

48.3 

32.2 

37.8 


Earned reward for ELA test b 

50.1 

45.8 

3.5 

41.4 

Earned reward for math test b 

63.2 

61.1 

6.9 

69.0 

Earned rewards for a Regents exam in 

0.0 

0.0 

51.5 

45.6 

Math 

0.0 

0.4 

33.9 

34.3 

Science 

0.0 

0.2 

31.8 

32.1 

Earned reward for earning at least 1 1 credits (%) 

0.0 

0.0 

54.6 

54.6 

Earned reward for obtaining a library card (%) 

65.6 

7.4 

- 

73.0 

Parent earned a reward for attending parent-teacher 
conference (%) 

61.7 

56.3 

45.1 

75.5 

Sample size 




823 


(continued) 
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Table 4.2 (continued) 


School Level and Outcome 

Year 1 

Year 2 

Year 3 

Years 1, 2, and 3 
Combined 

Students entering grade 9 





Earned any education reward (%) 

87.7 

76.4 

72.8 

92.0 

Total amount earned" ($) 

1,054 

1,217 

1,393 

3,117 

Earned at least 1 attendance reward (%) 

69.6 

56.2 

52.1 

76.7 

Earned an attendance reward in more than 4 periods (%) 

46.1 

37.1 

37.7 


Earned rewards for a Regents exam in 

36.1 

49.3 

58.6 

68.9 

Math 

24.1 

20.2 

9.5 

53.9 

Global History and Geography 

2.8 

26.9 

11.3 

41.1 

U.S. History and Government 

4.5 

4.3 

29.2 

37.9 

Comprehensive English 

2.2 

10.5 

34.2 

47.0 

Science 

23.8 

16.4 

9.8 

50.0 

Earned reward for earning at least 1 1 credits (%) 

51.4 

45.7 

45.5 

63.2 

Earned reward for obtaining a library card (%) 

59.9 

6.3 

- 

66.2 

Earned reward for taking the PSAT (%) 

0.0 

13.8 

10.4 

17.1 

Parent earned a reward for attending parent-teacher 





conference (%) 

52.6 

41.7 

33.4 

64.2 

Sample size 




988 


SOURCE: MDRC calculations using Seedco's Family Rewards program data. 

NOTES: Sample sizes may vary because of missing values. 

The first program year covers September 2007 through August 2008; the second program year 
covers September 2008 through August 2009; and the third program year covers September 2009 
through August 2010. 

A double dash (--) indicates "not applicable." 

a This is calculated only for students who earned any education rewards. 

b Receipt of elementary and middle school test score rewards in a given year is calculated only for 
students who are in grades that take the tests (grades 3 to 5 among elementary school students and 
grades 6 to 8 among middle school students). 


and the average amount earned, among those who did earn rewards, was $342 in Year 3. This 
fall in receipt is driven by several factors. First, the attendance rewards for elementary and 
middle school students were dropped in Year 3, given that attendance rates were fairly high 
already. Second, the new, higher standard for proficiency, mentioned earlier, meant that many 
fewer students earned the ELA and math test rewards. Finally, receipt of the reward for attend- 
ing a parent-teacher conference fell in Year 3. Although most parents reported meeting with 
their children’s teachers (shown later in this chapter), it is possible that they did not focus on 
submitting coupons for this reward, given that it represented only a $25 payment twice per year. 
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Reward receipt did not fall as dramatically for entering seventh-graders, and, on aver- 
age, they earned more in Year 3 than in earlier years. In Year 3, 82.4 percent of these students, 
most of whom were by then in ninth grade, earned a reward, and they earned an average of 
$1,203. The larger amount stems from the fact that these students were now eligible to earn 
rewards for passing Regents exams ($600 for each exam passed) and for accumulating 1 1 or 
more credits ($600 per year). Recall that much of the money earned for these rewards went to 
the students themselves, once they were in high school. For this reason, the program may have 
become more salient to these students in Year 3. 

The final panel of the table presents reward receipt for entering ninth-graders. Reward 
receipt is fairly stable from Year 2 to Year 3, as these students continued to earn rewards for 
passing Regents exams and earning 1 1 or more credits. On average, ninth-graders who partici- 
pated in the program and earned at least one reward earned more than $3,100 over the three- 
year program period. 


Impacts on School Activities and Outcomes 

This section presents the effects of the Family Rewards program on a variety of schooling 
outcomes. Data from the 42-month survey are used to present effects on intermediate outcomes, 
such as parental engagement with teachers and children, students’ use of tutoring, or students’ 
participation in extracurricular activities. Since most families were interviewed during the fall of 
the year after the program ended, data from the survey indicate whether the earlier impacts on 
parental engagement and student activities persisted once the program ended. Data from DOE 
records are used to present effects on school performance, including attendance rates, test 
scores, credits earned, and grade progression. These data cover four years after study entry, 
providing one year of post-program outcomes. 

Fourth-Grade Entry Cohort 

The supplement to this report presents data from the survey on various measures of pa- 
rental engagement and children’s perfonnance and activities for students who were entering 
fourth grade when they entered the study. 19 Given the timing of the survey, most of these 
students were in seventh grade when this survey was administered to their parents. At the 18- 


19 Parental engagement was measured by whether the respondent attended parent-teacher conferences, 
talked with the child about school, helped with homework, checked to make sure homework was complete, and 
helped the child prepare for tests. Children’s performance and activities were measured by child’s attendance at 
school, school performance (on a scale of 1 to 5), participation in extracurricular activities, and use of the 
Internet, a cell phone, and the library. See supplementary Appendix Table G.17 in Riccio et al. (2013), 
available at www.mdrc.org. 
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month point, the program led to small increases in self-reported parental engagement in their 
children’s schooling and an increase in use of the public library. At the 42 -month point, there 
are no effects on parental engagement, with teachers or with children, although these parents 
report a fairly high level of engagement already. For example, 95 percent of parents in the 
control group reported attending a parent-teacher conference since study entry. 

The survey data continue to show an increase in use of the public library — 87.4 per- 
cent of students in the program group were reported to have visited the library in the previous 
six months, compared with 82.7 percent of control group students. While obtaining a library 
card was rewarded, visiting the library was not, suggesting that visiting the library may have 
become habitual. One new effect that was not measured at the 18-month point was an increase 
in access to the Internet from home. Parents may have used the money from the program to buy 
a computer and pay for this access. 

Table 4.3 presents effects on enrollment status, attendance, and test scores based on 
DOE administrative records. This table focuses on the fourth-grade entry cohort. 20 MDRC’s 
earlier report on Family Rewards documented no effects for this group on these measures of 
school progress through Year 2. 21 The top panel of Table 4.3 shows that more than 90 percent 
of students in the control group who entered the study as fourth-graders were enrolled in the 
fifth grade during their second year in the program. In this case, enrollment indicates enrollment 
in a New York City public school. Most of the remaining students had transferred out of the 
school system, and a few students had been retained in the fourth grade. By the fourth year after 
study entry, only 86 percent of those students were enrolled in seventh grade. About 5 percent 
of those students were still in sixth grade, and most of the remaining students had transferred 
out of the New York City public school system during the four-year period. The program had 
no effects on enrollment status. 

Attendance rates were fairly high, at 9 1 percent on average for control group students in 
Year 1 . However, there is room for improvement in tenns of achieving very high attendance — 
only about 40 percent of those students attended school for 95 percent of the time or more time 
during each of the follow-up years. Attendance rates fall somewhat over the years, although that 
decrease is partly because nonenrolled students are included in the attendance measures. 22 
Recall from Figure 4.1 that attendance rates among enrolled students were fairly high, at more 
than 90 percent in each year. The program had no effects on attendance. 


20 Effects for all students who were in elementary school at study entry are similar to those presented here; 
see supplementary Appendix Table G.l in Riccio et al. (2013), available at www.mdrc.org. 

2l Riccio et al. (2010). 

~ 2 In order to maintain the experimental comparison, attendance impacts are calculated using the full pro- 
gram group and the full control group, even though some fraction of those students were no longer enrolled in 
New York City public schools in the later years of the follow-up period. 
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The Opportunity NYC Demonstration: Family Rewards 
Table 4.3 


Impacts on Enrollment, Attendance, and Test Scores, 
for Students in Grade 4 at the Time of Random Assignment 


Grade Level and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Enrollment status (%) 

Enrolled in grade 5, Year 2 

93.8 

94.7 

-0.9 

0.416 

Enrolled in grade 6, Y ear 3 

90.4 

92.0 

-1.6 

0.241 

Enrolled in grade 7, Y ear 4 

84.6 

85.5 

-0.9 

0.587 

Enrolled in any grade in Y ear 4 

90.3 

91.0 

-0.7 

0.620 

Attendance rate 95% or higher (%) 

Year 1 

43.2 

43.2 

0.0 

0.987 

Year 2 

44.5 

41.6 

2.9 

0.221 

Year 3 

41.3 

40.2 

1.1 

0.648 

Year 4 

40.0 

39.3 

0.7 

0.765 

Average attendance rate 1%) 

Year 1 

91.5 

91.0 

0.5 

0.417 

Year 2 

87.9 

88.3 

-0.4 

0.684 

Year 3 

84.6 

86.3 

-1.6 

0.187 

Year 4 

82.4 

82.7 

-0.3 

0.807 

Scored at proficient level or higher on ELA a (%) 

Year 1 

50.7 

51.1 

-0.4 

0.861 

Year 2 

67.6 

68.1 

-0.5 

0.816 

Year 3 

27.9 

29.1 

-1.3 

0.541 

Year 4 

24.8 

25.2 

-0.4 

0.846 

Scored at proficient level or higher on math a (%) 

Year 1 

73.4 

71.2 

2.1 

0.234 

Year 2 

80.3 

78.6 

1.7 

0.351 

Year 3 

40.5 

41.7 

-1.3 

0.561 

Year 4 

44.5 

43.1 

1.3 

0.555 

Sample size (total = 1,726) 

862 

864 




SOURCE: MDRC calculations using data from New York City Department of Education 
administrative records. 


NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and 
control groups. The p-value indicates the likelihood that the difference between the program and 
control groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 
percent; ** = 5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. Standard errors were adjusted to account for 
multiple observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Years 1, 2, 3, and 4 cover the 2007-2008, 2008-2009, 2009-2010, and 2010-201 1 school years, 
respectively. 

ELA = English language arts. 

a In New York State, students who score at a level of 3 or higher on a 4-point scale are deemed 
"proficient." 
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The final two panels in Table 4.3 present data on test scores. In contrast to the attend- 
ance measures, proficiency rates in each year were only calculated using students who took 
each test. However, few students were missing test score data in any year and there were 
minimal differences between the program and control groups in the rate of missing data. In 
Year 1, about half of entering fourth-graders in the control group were proficient on the ELA 
exam, and just over 71 percent were proficient on the math exam. As noted earlier, proficiency 
rates fell dramatically between Year 2 and Year 3, given the change in New York State stand- 
ards that are used to determine proficiency. 23 In Year 4, only one in four students in the study 
sample was deemed proficient in English language arts using the updated standards. The 
program had no effect on proficiency rates or average test scores (not shown). 

Seventh-Grade Entry Cohort 

The supplement to this report presents impacts on parental engagement and student ac- 
tivities for the group of students who were entering seventh grade when they entered the 
study. 24 Most of these students were in tenth grade when their parents responded to the 42- 
month survey. Data from the 18-month survey, presented in the last report, showed that the 
program led to a small increase in parents’ attendance at parent-teacher conferences, an increase 
in parents helping their children with homework, and an increase in these students’ participation 
in extracurricular activities. 25 

Data from the 42 -month survey show continued small effects on parents’ engagement 
with their children. Parents in the program group, compared with those in the control group, 
reported higher rates of talking with their children about school and checking their homework. In 
contrast, by Year 4, the program had no effects on extracurricular activities. Finally, as with the 
fourth-grade cohort, the program had a continued effect for this group on visits to the library. 

Table 4.4 presents effects on school progress using DOE administrative records. 26 Stu- 
dents in this group should have been in ninth grade in follow-up Year 3 and in tenth grade in 
follow-up Year 4, if they had been promoted regularly. Thus, the table also presents data in 
Year 3 and Year 4 on high school credits earned and Regents exams taken. To preserve the 
experimental comparison, these outcomes are presented for all students, with zero values for 


23 Although the primary change in test scoring was to raise the score needed for proficiency, the New York 
State Department of Education also revised the tests to make the questions less predictable. Average scores, as 
shown in Figure 4.1, fell very little between 2009 and 2010. 

24 See supplementary Appendix Table G.18 in Riccio et al. (2013), available at www.mdrc.org. 

25 Riccio et al. (2010). 

26 Effects for all middle school students are generally similar to those presented here and are shown in sup- 
plementary Appendix Table G.2 in Riccio et al. (2013), which is available at www.mdrc.org. 
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The Opportunity NYC Demonstration: Family Rewards 

Table 4.4 


Impacts on Enrollment, Attendance, Test Scores, Credits, and Regents 
Exams, for Students in Grade 7 at the Time of Random Assignment 


Grade Level and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Enrollment status (%) 

Enrolled in grade 8, Year 2 

95.0 

96.4 

-1.3 

0.179 

Enrolled in grade 9, Year 3 

87.6 

89.9 

-2.3 

0.145 

Enrolled in grade 10, Year 4 

70.3 

74.1 

-3.9 * 

0.076 

Enrolled in any grade in Y ear 4 

89.8 

91.7 

-1.9 

0.189 

Attendance rate 95% or higher (%) 

Year 1 

43.5 

43.0 

0.5 

0.846 

Year 2 

36.6 

35.0 

1.6 

0.477 

Year 3 

36.8 

34.3 

2.5 

0.287 

Year 4 

26.7 

24.9 

1.8 

0.402 

Average attendance rate (%) 

Year 1 

91.1 

90.8 

0.3 

0.533 

Year 2 

86.4 

87.6 

-1.2 

0.185 

Year 3 

79.3 

80.4 

-1.0 

0.462 

Year 4 

73.0 

74.5 

-1.5 

0.343 

Scored at proficient level or higher on ELA a (%) 

Year 1 

50.6 

50.6 

0.0 

0.995 

Year 2 

46.5 

46.0 

0.5 

0.809 

Scored at proficient level or higher on math a (%) 

Year 1 

60.4 

59.6 

0.8 

0.675 

Year 2 

61.9 

63.5 

-1.6 

0.429 

Attempted 11+ credits (%) 

Year 3 

78.0 

79.0 

-0.9 

0.621 

Year 4 

77.3 

77.2 

0.1 

0.963 

Earned 11+ credits (%) 

Year 3 

50.9 

50.7 

0.2 

0.931 

Year 4 

46.5 

48.5 

-2.0 

0.403 

Number of Regents exams taken 

Year 3 

1.2 

1.1 

0.0 

0.400 

Year 4 

1.9 

1.9 

0.0 

0.836 

Number of Regents exams passed 

Year 3 

0.7 

0.7 

0.0 

0.293 

Year 4 

1.0 

1.0 

0.1 

0.261 

Sample size (total = 1,670) 

823 

847 




(continued) 
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Table 4.4 (continued) 

SOURCE: MDRC calculations using data from New York City Department of Education 
administrative records. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the difference between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 
5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. Standard errors were adjusted to account for multiple 
observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Years 1, 2, 3, and 4 cover the 2007-2008, 2008-2009, 2009-2010, and 2010-2011 school years, 
respectively. 

The Regents measures in this table include the following Regents exams: English, Math A, 

Math B, Geometry, Integrated Algebra, Algebra 2/Trigonometry, U.S. History and Government, 
Global History and Geography, Living Environment, Chemistry, Physics, and Earth Science. 

a ln New York State, students who score at a level of 3 or higher on a 4-point scale are deemed 
"proficient." 


students who had not been promoted to high school by Year 3 or Year 4. The previous report 
documented few effects on attendance and test scores for this cohort. 

Looking at the control group in Table 4.4, under “Enrollment status,” the top rows show 
that 96.4 percent of entering seventh-graders were enrolled as eighth-graders in Year 2, and 89.9 
percent were enrolled as ninth-graders in Year 3. Progress slows dramatically after Year 3, 
however, or during the transition between ninth and tenth grade. By Year 4, only 74. 1 percent of 
entering seventh-graders in the control group are enrolled in tenth grade, representing a 16 
percentage point drop in on-grade status from the preceding year. A majority of the remaining 
26 percent of students were still in ninth grade (15 percent). Most of the other students had 
transferred out of the school system during the four-year period. The program appears to have 
led to a small reduction (when comparing the program and control groups), of 3.9 percentage 
points, in on-grade status in Year 4. However, the lack of effects on other measures of school 
progress suggests that this difference may be random variation and not a true program effect. 

Attendance rates fall similarly over the years, with only 25 percent of students in the 
control group attending at the 95 percent rate or higher in Year 4. Recall that nonenrolled 
students are included in these measures with zero values, which partly explains the low average 
rates. When these students are taken out of the calculation, as shown in Figure 4.1, the average 
rate increases to 82 percent in Year 4. The program had no effect on attendance rates. The sub- 
sequent rows present data on test scores and the high school outcomes of credits earned and 
Regents exams passed; the program had no effect on those outcomes. 
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Ninth-Grade Entry Cohort 

The supplement to this report presents impacts on parental engagement and student ac- 
tivities for students who entered the study as they were entering ninth grade. 27 According to 
DOE records, about half of entering ninth-graders were in twelfth grade when their parents took 
the survey; 28 percent were still enrolled in ninth, tenth, or eleventh grade; and 2 1 percent were 
no longer enrolled (not shown). The last report showed that the program had no effect on 
parental engagement at the 18-month point and a small effect on extracurricular activities. 28 

The 42 -month data show a new, small effect on parental engagement. Parents in the 
program group were more likely to report checking their children’s homework and talking with 
their children about school. The program did have some effects, albeit small, on ninth-graders’ 
progress during the three-year program period, as measured by the DOE records data. It may be 
that parents in the program group were responding to their children’s improved performance by 
becoming more engaged. 

As shown in Appendix Table G.19 in the supplement to this report, 29 almost 92 percent 
of parents in the control group reported that their children were enrolled in school. These reports 
match the DOE data fairly well for the subgroup of students whose parents responded to the 
survey. 30 The program did not lead to an increase in reported enrollment. The difference of 2.8 
percentage points between the program and control groups is not statistically significant. 

Finally, the program led to an increase in participation in extracurricular activities, which 
is driven largely by an increase in programs to help with school work and participation in 
sports. 31 Program group students were also more likely to have a library card and to report 
having a cell phone. 32 


27 See supplementary Appendix Table G.19 in Riccio et al. (2013), available at www.mdrc.org. 

28 Riccio et al. (2010). 

29 See Riccio et al. (2013), available at www.mdrc.org. 

30 DOE data indicate that about 88 percent of these children (of survey respondents) were enrolled in 
school during Year 4. When parents were asked where their children were enrolled, 88 percent reported a New 
York City public school. The remaining 3 percent were enrolled in private schools in New York City or in 
schools outside New York City. 

3 ’Although the survey did not ask parents whether the school offered these sports, participation in school- 
sponsored sports does require minimum attendance and credits; see the Public School Athletic League Web 
site, www.psal.org. It is possible that Family Rewards helped more students become eligible for school sports 
by increasing their attendance and credits earned in earlier years. 

32 The survey also collected parents’ reports on whether youth were involved in a variety of risky behav- 
iors, including arrests, trouble with the police, substance use, and childbearing. The program had no effects on 
those outcomes (not shown). 
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Table 4.5 presents effects on school progress using DOE administrative records. 33 The 
previous report documented small effects on attendance and effort through Year 2 for the full 
sample of entering ninth-graders. 34 Students in the program group attended at higher rates and 
they attempted more credits and Regents exams, although they did not earn more credits or pass 
more Regents exams than their control group counterparts. 

As noted earlier, only 5 1 percent of the control group students who had entered ninth 
grade at the start of the study were enrolled in twelfth grade by Year 4, although 79 percent of 
them were still enrolled in school. Most of the remaining students had either dropped out or 
transferred in an earlier year. 

The table illustrates that the only lasting effect of the program for this cohort is an in- 
crease in the number of Regents exams they had taken by the end of Year 4. Through Year 4, 
students in the program group took an average of 5.9 Regents exams, compared with 5.6 exams 
for the control group. However, there was no effect on the percentage of students who had 
passed five exams by the end of Year 4, the number needed to obtain a Regents diploma. 

The program had no effect on enrollment status but did increase the likelihood of very 
high attendance in Years 2 and 3. In Year 3, 25.1 percent of program group students attended 
school at least 95 percent of the time, compared with 21.9 percent of control group students, for a 
statistically significant difference of 3.1 percentage points. Similarly, the program did not affect 
credits earned overall, with the exception of a small increase in Year 3. However, by the end of 
Year 4, when the rewards were no longer offered, the number of students with 44 credits or more 
was similar for the two groups. The program also did not increase the graduation rate. 


Impacts for Key Subgroups 

The previous report on Family Rewards presented program effects for several subgroups that 
were selected in advance of the analysis — students’ prior perfomiance, parents’ education 
level, and students’ school environment. 35 The results showed little variation in effects across 


33 Results for the full group of entering high school students are shown in supplementary Appendix Table 
G.3 in Riccio et al. (2013), which is available at www.mdrc.org. 

34 Riccio et al. (2010). 

35 Effects across additional dimensions were also examined as part of a more exploratory analysis. These 
other dimensions include baseline poverty level, public assistance receipt, public housing status, parents’ 
reported risk of depression, parents’ employment, family structure, immigration status, and gender. In general, 
the results showed little variation in effects across these groups. The analysis is more exploratory in nature, 
meaning that the findings are less certain, given that these subgroups were not identified before the start of the 
analysis, and the likelihood of finding effects due to chance increases as the number of subgroups that are 
analyzed increases. These results are presented in supplementary Appendix Tables G.6 through G.14 in Riccio 
et al. (2013), available at www.mdrc.org. 
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The Opportunity NYC Demonstration: Family Rewards 

Table 4.5 


Impacts on Enrollment, Graduation, Attendance, Credits, and Regents 
Exams, for Students in Grade 9 at the Time of Random Assignment 



Program 

Control 

Difference 


Grade Level and Outcome 

Group 

Group 

(Impact) 

P- Value 

Enrollment and graduation (%) 

Enrolled in grade 10, Year 2 

73.1 

70.6 

2.6 

0.193 

Enrolled in grade 11, Year 3 

54.8 

53.0 

1.7 

0.411 

Enrolled in grade 12, Year 4 

53.1 

51.2 

1.9 

0.360 

Enrolled in any grade in Y ear 4 

80.1 

79.2 

0.9 

0.610 

Graduated within 4 years 

49.2 

48.2 

1.1 

0.621 

Attendance rate 95% or higher (%) 

Year 1 

34.0 

31.5 

2.5 

0.211 

Year 2 

28.8 

23.7 

^ J H=H=H= 

0.007 

Year 3 

25.1 

21.9 

3.1 * 

0.089 

Year 4 

17.4 

15.3 

2.1 

0.197 

Average attendance rate (%) 

Year 1 

81.8 

81.4 

0.4 

0.683 

Year 2 

75.3 

74.3 

1.0 

0.439 

Year 3 

69.4 

67.7 

1.7 

0.254 

Year 4 

60.7 

59.7 

1.1 

0.508 

Attempted 11+ credits (%) 

Year 1 

87.8 

83.9 

3 g *** 

0.006 

Year 2 

80.5 

77.9 

2.6 

0.126 

Year 3 

71.0 

68.0 

3.0 

0.126 

Year 4 

45.6 

47.4 

-1.9 

0.403 

Earned 11+ credits (%) 

Year 1 

49.7 

50.0 

-0.3 

0.896 

Year 2 

45.2 

45.4 

-0.2 

0.928 

Year 3 

42.8 

39.2 

3.7 * 

0.080 

Year 4 

31.6 

31.5 

0.1 

0.961 

Earned at least 44 credits, Y ears 1 to 4 

41.5 

40.5 

0.9 

0.652 

Average number of credits earned, Y ears 1 to 4 

32.7 

31.9 

0.8 

0.300 

Regents exams. Years 1 to 4 

Number taken 

5.9 

5.6 

0.3 * 

0.085 

Number passed 

3.0 

2.9 

0.1 

0.123 

Passed at least 5 exams (%) 

36.7 

35.7 

1.1 

0.562 

Sample size (total = 1,978) 

988 

990 
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Table 4.5 (continued) 


SOURCE: MDRC calculations using data from New York City Department of Education 
administrative records. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and 
control groups. The p-value indicates the likelihood that the difference between the program and 
control groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 
percent; ** = 5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. Standard errors were adjusted to account for 
multiple observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

The Regents measures in this table include the following Regents exams: English, Math A, 
Math B, Geometry, Integrated Algebra, Algebra 2/Trigonometry, U.S. History and Government, 
Global History and Geography, Living Environment, Chemistry, Physics, and Earth Science. 

Years 1, 2, 3, and 4 cover the 2007-2008, 2008-2009, 2009-2010, and 2010-2011 school 
years, respectively. 


subgroups for the elementary and middle school students. The updated results through Year 4 
tell a similar story and are presented in Appendix Tables C. 1 to C.6. 36 

For the ninth-graders, the effects of the program also did not differ significantly by par- 
ents’ education level or by school environment. 37 School environment was defined using test 
scores of earlier cohorts in the school a student entered when he or she entered the study. Specifi- 
cally, students’ schools were ranked according to their average pass rates for the English and 
math Regents exams in the 2005-2006 and 2006-2007 school years. The schools were then 
divided into thirds based on this ranking. As shown in Appendix Table C.8, the effects of the 
program did not vary across school types. 

As another test of school environment, students were divided into two groups, based on 
whether a student did or did not enter one of the new small high schools that were opened in 
New York City over the past decade in response to the closing of many large, failing high 
schools. In an ongoing evaluation by MDRC that takes advantage of the lottery-like process by 
which students are admitted to New York City high schools, these small schools have been 
found to lead to large improvements in school progress and graduation rates. 38 For this reason, 


36 There is some suggestion of varying effects by school environment for entering fourth- and seventh- 
graders, with negative effects on attendance at the lowest-ranking schools and positive effects at the highest- 
ranking schools. However, this pattern does not carry over to test scores. In addition, a similar pattern is not 
found for entering ninth-graders. 

37 These results are presented in Appendix Tables C.7 and C.8. 

38 Bloom and Unterman (2012). 
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these small schools are arguably “higher quality” than other schools. However, the effects of 
Family Rewards were not significantly different for students in these small schools compared 
with students in other schools. 39 

In contrast, as shown in the previous report on Family Rewards, the effects of the pro- 
gram did differ by students’ level of academic proficiency when they entered the study. Table 
4.6 presents four-year effects by math proficiency, or for students who scored at the proficient 
level on the eighth-grade math test versus students who did not score at the proficient level. The 
findings show large positive effects through Year 3 for the proficient group, and few effects for 
the nonproficient group. For example, in Year 3 the program led to an increase of 6.7 percent- 
age points in the percentage of students who earned 1 1 or more credits, an impact that is similar 
in size to the impact on the number of credits attempted. However, by the end of Year 4, the 
program group was not more likely to have earned at least 44 credits and not more likely to 
have graduated. This lack of effect seems to be in part the result of a negative effect on credits 
attempted in Year 4, in which program group students were 8.3 percentage points less likely to 
have attempted 1 1 or more credits. 

A key question in the Family Rewards evaluation is whether any observed behavioral 
responses to the incentives, such as attempting more credits or taking more Regents exams, 
would end once the incentives were taken away. The data in Table 4.6 suggest that effects on 
attendance and credits attempted did diminish after the incentives were removed, although it 
is somewhat surprising that the effects were negative for credits attempted. The fact that the 
program and control groups were equally likely to have earned at least 44 credits by the end 
of Year 4 might suggest that most of this reduction in attempted credits represents a dimin- 
ished effort in Year 4 by program group students because they had earned more than 33 
credits by the end of Year 3. However, a reduction in motivation because the incentives were 
removed cannot be ruled out. 40 

The last panel of Table 4.6 presents data on rewards earned for the two groups. Not sur- 
prisingly, students who were math-proficient earned almost twice the amount of rewards that 
their less proficient counterparts earned, approximately $4,500 versus $2,400, with the biggest 
difference in amounts earned for Regents exams. 


39 These results are presented in supplementary Appendix Table G.15 in Riccio et al. (2013), which is 
available at www.mdrc.org. 

40 Although it is a nonexperimental comparison, a similar reduction in attempting 1 1 credits or more in 
Year 4 was observed among those students who had accumulated less than 33 credits through Year 3, and thus 
would have needed at least 1 1 more credits to graduate. This result is consistent with findings from a program 
for elementary school students that rewarded students for achievement on one reading and math test per year. 
Among students in the program group who had already earned the maximum incentive from the first test, the 
effect of the program was to reduce their performance on the second test; see Bettinger (2010). 
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The Opportunity NYC Demonstration: Family Rewards 
Table 4.6 


Impacts on Enrollment, Graduation, Attendance, Credits, and Regents Exams, for Students in Grade 9 
at the Time of Random Assignment, by Performance on Math Test in the Prior Year (Grade 8) 



Proficient on 
8th Grade Math Test a 

Not Proficient on 
8th Grade Math Test 1 


Subgroup and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Sig. 

Enrollment and graduation (%) 

Enrolled in grade 12, Year 4 

77.6 

71.8 

5.8 

47.5 

47.0 

0.5 


Enrolled in any grade in Year 4 

91.1 

89.3 

1.8 

81.9 

80.5 

1.4 


Graduated within 4 years 

74.8 

71.3 

3.5 

41.7 

42.2 

-0.5 


Attendance rate 95% or higher (%) 

Year 1 

53.8 

46.9 

7.0 * 

29.0 

26.9 

2.1 


Year 2 

50.8 

36.7 

14 \ *** 

21.8 

19.4 

2.4 

tt 

Year 3 

42.6 

34.4 

8.2 ** 

19.2 

18.1 

1.0 


Year 4 

28.8 

23.2 

5.6 

14.1 

13.2 

0.8 


Average attendance rate (%) 

Year 1 

91.4 

88.1 

3.3 ** 

82.2 

81.5 

0.6 


Year 2 

87.9 

82.9 

5 j *** 

74.0 

74.0 

0.1 

tt 

Year 3 

82.8 

78.2 

4.6 ** 

67.8 

66.7 

1.1 


Year 4 

75.6 

72.6 

3.0 

59.2 

58.9 

0.3 


Attempted 11+ credits (%) 

Year 1 

95.7 

91.5 

4.2 ** 

90.5 

84.9 

5.6 *** 


Year 2 

92.3 

89.5 

2.9 

80.8 

78.8 

2.1 


Year 3 

86.9 

81.1 

5.9 * 

71.2 

69.0 

2.1 


Year 4 

51.3 

59.7 

-8.3 ** 

48.3 

47.2 

1.1 

t 
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Table 4.6 (continued) 



Proficient on 
8th Grade Math Test' 1 

Not Proficient on 
8th Grade Math Test a 


Subgroup and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Sig. 

Earned 11+ credits (%) 

Year 1 

77.4 

68.9 

8.5 ** 

43.5 

47.1 

-3.6 

ttt 

Year 2 

71.5 

62.8 

g y ** 

37.6 

41.5 

-3.9 

ttt 

Year 3 

64.2 

57.5 

6.7 * 

37.3 

35.0 

2.3 


Year 4 

42.2 

44.6 

-2.4 

30.6 

29.4 

1.2 


Earned at least 44 credits, Years 1 to 4 (%) 

64.4 

60.4 

4.1 

36.9 

36.8 

0.1 


Average number of credits earned, Years 1 to 4 

43.5 

41.3 

2.3 * 

30.8 

30.6 

0.2 


Resents exams. Years 1 to 4 

Number taken 

7.3 

7.0 

0.3 

5.8 

5.5 

0.3 


Number passed 

5.1 

4.9 

0.2 

2.3 

2.2 

0.1 


Passed at least 5 exams (%) 

69.5 

68.8 

0.7 

25.6 

24.0 

1.6 


Rewards earned. Years 1 to 4 ($) 

Total amount earned 

4,490 

— 

— 

2,369 

— 

— 


Amount earned from attendance 

815 

- 

- 

460 

— 

— 


Amount earned from Regents exams 

2,313 

— 

— 

1,168 

— 

— 


Amount earned from earning 11+ credits 

1,297 

— 

— 

745 

— 

— 


Sample size (total = 1,726) 

298 

285 


565 

578 
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Table 4.6 (continued) 

SOURCE: MDRC calculations using data from New York City Department of Education administrative records. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. The p-value indicates the 
likelihood that the difference between the program and control groups arose by chance. Statistical significance levels are indicated as follows: 
*** = i percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical significance levels (Sig.) are indicated as follows: 
tft = 1 percent; ft = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. 
Standard errors were adjusted to account for multiple observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Years 1, 2, 3, and 4 cover the 2007-2008, 2008-2009, 2009-2010, and 2010-201 1 school years, respectively. 

The Regents measures in this table include the following Regents exams: English, Math A, Math B, Geometry, Integrated Algebra, Algebra 
2/Trigonometry, U.S. History and Government, Global History and Geography, Living Environment, Chemistry, Physics, and Earth Science. 

A double dash (--) indicates "not applicable." 

a In New York State, students who score at a level of 3 or higher on a 4-point scale are deemed "proficient." 



Proficiency status for the previous report on Family Rewards was defined using the 
math test. Results using the ELA test were not shown because they were fairly similar to the 
findings using math through Year 2. However, the additional data through Year 4 show 
emerging differences between the two proficiency groups. Table 4.7 presents the results by 
ELA proficiency. 

For these students, as for the math-proficient students, the effects on attendance and 
credits attempted diminish after Year 3, when the incentives ended. However, the large early 
gains translated into benefits that lasted through the end of Year 4. For example, students in the 
program group, compared with the control group, were 10 percentage points more likely to be 
enrolled in twelfth grade in Year 4 and 10 percentage points more likely to have earned at least 
44 credits by the end of that year. The program also led to notable effects on the number of 
Regents exams passed and on graduation rates. The graduation rate for students in the control 
group, for example, was 66.9 percent, compared with 74.8 for students in the program group, an 
increase of 8 percentage points, or 12 percent. These effects are on a par with graduation effects 
observed in the recent evaluation ofNew York City’s new small schools. 41 

Finally, a separate study of incentives (discussed below) found notable differences by 
gender, with positive effects only for girls. To examine whether this pattern holds for Family 
Rewards, effects were examined for ELA-proficient boys versus ELA-proficient girls. The effects 
of the program are substantially larger and more consistent for girls than for boys. 42 However, 
none of the differences in impacts between boys and girls was statistically significant, owing 
largely to the small sample size. Thus, it is not possible to assert with any degree of certainty 
that the program worked better for proficient girls. Nonetheless, the results are intriguing and 
warrant further study and replication. 

In sum, the results by proficiency status indicate that the program had positive effects 
on students who entered the study more academically prepared than their peers. The program 
encouraged those students to increase their efforts (attendance and attempted credits), which led 
to an increase in the number of them who reached key milestones (Regents exams passed and 
graduation). Findings from a Family Rewards child and family study also illustrate how the 
program encouraged additional effort from these proficient students. 43 


41 Bloom and Unterman (2012). 

42 These results are presented in supplementary Appendix Table G.16 in Riccio et al. (2013), which is 
available at www.mdrc.org. 

43 Specifically, the program changed the ways in which teenagers spent their time while they were not in 
school, increasing the proportion of teenagers who engaged primarily in academic activities — for example, 
homework and achievement-oriented after-school activities. These effects were concentrated among proficient 
ninth-graders. See Morris, Aber, Wolf, and Berg (2012.) 
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The Opportunity NYC Demonstration: Family Rewards 

Table 4.7 

Impacts on Enrollment, Graduation, Attendance, Credits, and Regents Exams, 
for Students in Grade 9 at the Time of Random Assignment, by Performance 
on English Language Arts (EL A) Test in the Prior Year (Grade 8) 

Proficient on Not Proficient on 

8th Grade ELA Test a 8th Grade ELA Test a 



Program 

Control 

Difference 

Program 

Control 

Difference 


Subgroup and Outcome 

Group 

Group 

(Impact) 

Group 

Group 

(Impact) 

Sig. 

Enrollment and graduation (%) 

Enrolled in grade 12, Year 4 

78.4 

68.2 

j *** 

48.3 

50.3 

- 2.0 

tt 

Enrolled in any grade in Y ear 4 

90.9 

89.0 

1.9 

82.7 

81.3 

1.4 


Graduated within 4 years 

74.8 

66.9 

8.0 ** 

43.2 

45.9 

- 2.8 

tt 

Attendance rate 95% or higher (%) 

Year 1 

54.9 

42.3 

12.6 *** 

29.6 

29.6 

0.1 

tt 

Year 2 

47.0 

33.9 

j3 j *** 

24.3 

21.5 

2.8 

tt 

Year 3 

40.3 

30.8 

9.5 ** 

21.4 

20.4 

1.0 

t 

Year 4 

28.1 

23.0 

5.2 

14.8 

14.3 

0.5 


Average attendance rate (%) 

Year 1 

92.0 

86.9 

^ j *** 

82.1 

82.3 

- 0.2 

ttt 

Year 2 

87.5 

81.4 

0 j *** 

74.9 

75.1 

-0.3 

tt 

Year 3 

83.9 

77.1 

6.8 *** 

68.1 

67.7 

0.4 

tt 

Year 4 

76.7 

71.6 

5.1 * 

59.8 

60.4 

- 0.6 


Attempted 11+ credits (%) 

Year 1 

95.8 

91.6 

4.2 ** 

90.7 

85.5 

^ j *** 


Year 2 

93.0 

86.3 

6.6 ** 

80.9 

80.4 

0.4 

t 

Year 3 

87.7 

80.3 

74 ** 

71.8 

70.1 

1.7 


Year 4 

53.3 

58.8 

-5.5 

48.3 

48.8 

-0.5 
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Table 4.7 (continued) 



Proficient on 
8 th Grade ELA Test 3 

Not Proficient on 
8 th Grade ELA Test 3 


Subgroup and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Sig. 

Earned 11+ credits (%) 

Year 1 

76.3 

66.1 

10.2 *** 

45.0 

49.6 

-4.6 

ttt 

Year 2 

71.3 

58.5 

Y2 9 *** 

38.0 

44.5 

-6.5 ** 

ttt 

Year 3 

65.1 

53.4 

1 1 g *** 

37.8 

37.9 

-0.1 

tt 

Year 4 

42.8 

43.8 

-1.0 

31.1 

31.0 

0.1 


Earned at least 44 credits, Y ears 1 to 4 (%) 

66.1 

56.6 

9.6 ** 

36.9 

39.8 

-2.8 

tt 

Average number of credits earned, Years 1 to 4 

44.3 

40.0 

4 3 *** 

30.9 

31.8 

-0.8 

ttt 

Resents exams. Years 1 to 4 

Number taken 

7.2 

6.5 

0 g *** 

5.9 

5.8 

0.1 

t 

Number passed 

5.0 

4.6 

0.5 ** 

2.4 

2.5 

0.0 

tt 

Passed at least 5 exams (%) 

72.5 

63.1 

9.5 ** 

25.8 

28.9 

-3.1 

ttt 

Rewards earned. Years 1 to 4 ($) 

Total amount earned 

4,536 

— 

— 

2,447 

— 

— 


Amount earned from attendance 

800 

— 

— 

483 

— 

— 


Amount earned from Regents exams 

2,342 

- 

— 

1,203 

- 

- 


Amount earned from earning 11+ credits 

1,308 

— 

— 

764 

- 

- 


Sample size (total = 1,700) 

271 

256 


576 

597 
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Table 4.7 (continued) 


SOURCE: MDRC calculations using data from New York City Department of Education administrative records. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. The p-value indicates the 
likelihood that the difference between the program and control groups arose by chance. Statistical significance levels are indicated as follows: 
*** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical significance levels (Sig.) are indicated as follows: 
fff = 1 percent; f f = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random assignment characteristics of sample 
members. Standard errors were adjusted to account for multiple observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Years 1, 2, 3, and 4 cover the 2007-2008, 2008-2009, 2009-2010, and 2010-2011 school years, respectively. 

The Regents measures in this table include the following Regents exams: English, Math A, Math B, Geometry, Integrated Algebra, 
Algebra 2/Trigonometry, U.S. History and Government, Global History and Geography, Living Environment, Chemistry, Physics, and Earth 
Science. 

A double dash (--) indicates "not applicable." 

a ln New York State, students who score at a level of 3 or higher on a 4-point scale are deemed "proficient." 



As mentioned in the previous report, given that families were largely left on their own 
to find ways to earn the incentives in Family Rewards, it makes sense that the achievement 
gains would be larger for the students who were better prepared academically. These students 
were staying afloat academically and probably had the personal and other resources necessary 
to take advantage of the incentives offered. It is not clear why the effects would be larger for 
students who are proficient in ELA. However, given that reading and language skills are 
relevant and necessary for all school subjects, it is easy to imagine that these students were the 
most prepared to respond to the program . 44 

The findings are consistent with those from several other studies of incentives in educa- 
tion, where effects were largest for students who were on the threshold of achievement, or those 
for whom the desired outcomes are within reach. The most similar program is one in Israel that 
offered incentives to high school students for taking and passing matriculation exams and for 
graduating . 45 That program led to positive effects on graduation rates and college enrollment 
rates. However, the effects occurred largely for girls, and for the subset of girls who were more 
academically prepared when they entered the study. 

Two other programs targeted different age groups but found similar effects. A program 
in Ohio, which offered incentives to elementary school students for good test perfonnance, had 
its largest effects on students who had higher test scores before they entered the study . 46 
Another program that offered incentives to first-year college students had positive effects on 
school progress for the most prepared students and negative effects for the least prepared 
students . 47 

Finally, as an additional test of the robustness of the results, effects by proficiency status 
were estimated separately for ninth-graders in each of the three New York boroughs, based on 
the argument that those boroughs can be considered three separate and independent tests of the 
program. The findings were very similar across the three places. 


Conclusion 

Through four years, Family Rewards has had mixed success in improving children’s school 
outcomes. The program had few effects on younger students, but did lead to notable gains — an 


44 Many of the same students who are proficient in ELA are also proficient in math, although the two out- 
comes are not perfectly correlated. For example, among those students who are proficient in ELA, about 60 
percent are also proficient in math. 

45 Angrist and Lavy (2009). 

46 Bettinger (2010). 

47 Leuven, Oosterbeek, and van der Klaauw (2010). 
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increase in the graduation rate of 8 percentage points — for reading-proficient ninth-graders. 
Even though this group may be the least disadvantaged academically among low-income 
students, they still face many obstacles to success. An increase in graduation rates of this 
magnitude is impressive and on a par with effects from other, more intensive interventions that 
change the school structure, for example, or alter instructional practices. 

The question remains why the program did not affect performance for other students — 
for example, younger students and nonproficient ninth-graders. Despite early, albeit small, 
increases in parental engagement and sizable reductions in poverty and material hardship, 
Family Rewards had no effect on the school progress of elementary and middle school students. 
In temis of the effectiveness of the incentives, perhaps they were not as salient to the younger 
children, since these children did not receive the money themselves. Also, many of the parents 
of these younger students may have wanted to help their children earn the incentives but did not 
know what steps to take. 48 

The less proficient ninth-graders may have similarly not known how to improve their 
grades or test scores. The replication of Family Rewards through the Social Innovation Fund, 
discussed in more detail in Chapter 7, seeks to address this issue. Unlike the original version of 
the program that is described in this report, in which staff were deliberately constrained in the 
services they could provide, staff in the replication program are charged with working proac- 
tively with families to help them earn the rewards, which may include directing these families to 
tutoring and other educational services when their children are struggling academically. 

Finally, one question about the education rewards from the outset was whether the be- 
havioral effects would end once the program ended or, worse, whether they would become 
negative if the program had somehow reduced students’ intrinsic motivation levels. Data for the 
ninth-graders indicate that there was a small reduction in credits attempted in Year 4, although it 
is not clear that this effect represents reduced intrinsic motivation. In addition, the effects on 
attendance did diminish somewhat after Year 3. For proficient ninth- graders, the fading effects 
may not have mattered much, since the program’s effect in helping them reach twelfth grade 
may have been enough of a gain to sustain them through that year without rewards (although 
the program did offer a graduation reward). For younger students, however, the rewards may 
need to be structured in a different way in order to have lasting effects. 


48 The Spark program, evaluated as part of Opportunity NY C, also provided incentives to fourth- and sev- 
enth-graders for achieving certain minimum test scores. However, that program was found to have no effects 
on test scores or other measures of school progress; see Fryer (2011). Fryer suggests that these students did not 
know how to improve their school progress in order to earn the incentives and that the incentives would be 
more effective if they were provided for inputs to school progress (such as reading or homework), rather than 
outputs (such as test scores). 
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Chapter 5 

Effects on Health Care and Health 


Making some safety net benefits for low-income families conditional on certain preventive 
health care practices is a feature of conditional cash transfer (CCT) programs worldwide. These 
programs typically require families to obtain regular checkups at health clinics and, in some cas- 
es, to participate in health and nutrition education sessions that are designed to promote active 
attention to health care in order to receive cash benefits. 1 Family Rewards incorporated this prin- 
ciple of linking financial payments to preventive health care but adapted it for the very different 
health care and safety net context found in the United States. It offered incentives to low-income 
families to maintain public or private health insurance and to obtain age-appropriate preventive 
medical and dental checkups. 2 

This chapter describes the health impacts of Family Rewards. It presents findings on the 
longer-term effects of the program on participants’ health insurance coverage, use of health care 
services, and health outcomes. The findings show that at around 42 months after random as- 
signment, when participating parents were interviewed, Family Rewards produced a number of 
small, positive effects on outcomes in the health domain. It increased the continuity of health 
insurance coverage and increased the receipt of regular preventive care, particularly dental care. 
It may have produced a few beneficial effects on health outcomes for certain adult populations. 
However, other outcomes of interest were unaffected by the intervention, and, as noted below, a 
number of impacts that were observed during the previous survey period (18 months after ran- 
dom assignment) had dissipated by the time of the 42 -month follow-up. Although most of the 
health effects of Family Rewards are small and limited, they are observed on outcomes that are 
generally considered difficult to influence through social interventions. 3 


Fernald. Gertler, andNeufeld (2008). 

2 See Chapter 1 and Riccio et al. (2010) for a discussion of the logic model and the pathways through 
which Family Rewards is expected to influence health. 

3 The literature on the effectiveness — or the relative value — of preventive clinical interventions (that is, 
interventions designed to change health care behavior) is vast and inconsistent. In general, the response to pre- 
ventive clinical interventions is weak. In a systematic review and analysis of recent interventions, Maciosek et al. 
(2006) find insufficient evidence to support the effectiveness of a variety of counseling and preventive care prac- 
tices (for example, counseling the general population of adults and children about physical activity and diet; 
counseling children and adults about preventive dental care practices; or counseling older children, adolescents, 
and adults on safety practices). Furthermore, while some evidence suggests that health insurance alone has bene- 
ficial effects on life expectancy, the impacts are small; see Muennig, Franks, and Gold (2005). On the other 
hand, the effects of social interventions, such as improved early childhood education, have demonstrated effec- 
tive ways of improving later health outcomes; see Belfield (2007). The gains are large, ranging from four to nine 

(continued) 
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As described in the early evaluation report, the families in the Family Rewards sample 
had higher-than-expected insurance coverage and patterns of preventive care when they entered 
the study. 4 These characteristics may reflect efforts that began around 2000 in New York City 
and New York State to improve access to health care coverage and to improve the health deliv- 
ery system for low- income and moderate- income families. For example, state and city officials 
have worked with a range of stakeholders to implement measures to reduce administrative bar- 
riers to enrollment in and maintenance of public health insurance programs in order to provide 
coverage to more individuals. 5 Moreover, New York City has been at the forefront of creating 
public-private partnerships to identify and enroll eligible uninsured residents. This enrollment 
push has been aided by a combination of state funding support, city agency partnerships, and 
use of community-based enrollment staff. 6 New York City and New York State have also en- 
acted a series of reforms to increase the retention of public insurance among participating fami- 
lies and individuals. 7 For instance, they have reduced documentation requirements for Medicaid 
renewal and streamlined the renewal process. 

All these reforms are an important backdrop for understanding the health-related im- 
pacts of Family Rewards. They are likely to account for the already high health care coverage 
and preventive health care practices reported by control group parents at the time they entered 
the study and in follow-up interviews. Consequently, on some measures, Family Rewards had 
limited room to improve health outcomes further. 


Measuring the Health -Related Impacts of Family Rewards 

The analysis of health-related outcomes and impacts is based on a variety of data sources. First, 
Seedco’s Family Rewards payment system provides data on the receipt of health rewards during 
the program term. As described in Chapter 2, this system records the number and types of 
health reward payments earned, as well as the total amount earned by parents and children. Sec- 
ond, the analysis of public health insurance coverage (that is, Medicaid) is based on administra- 
tive records obtained from New York City’s Human Resources Administration (HRA). 8 Those 


years in increased life expectancy for those who earn a high school diploma, or up to an 11.5 percent increase; 
see Muennig (2000). These interventions, however, require many years to demonstrate effects. 

4 Riccio et al. (2010). 

5 However, undocumented immigrants are not eligible for public health insurance because their immigra- 
tion status disqualifies them for the programs. 

6 New York City Mayor’s Office of Health Insurance Access (2004). 

7 See United Hospital Fund (2009). 

8 As noted in Chapter 1 of Riccio et al. (2010), Medicaid is available to pregnant women and children under 
6 years of age whose family income is at or below 133 percent of the federal poverty level, for children ages 6 to 
19 with family income up to 100 percent of the federal poverty level, and for families receiving government 

(continued) 
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data are used to calculate public health insurance receipt for parents and children, and are avail- 
able for one quarter before random assignment and for three years afterward for the full sam- 
ple. 9 Third, the Family Rewards 42 -month parent survey provides extensive information on 
health care practices and outcomes that cannot be captured from the administrative records data- 
bases. The 42-month survey was administered to a random subset of the full study sample, 10 and 
it is the only source of data for analyzing the program’s effects on parents’ and children’s health 
care use and health outcomes. It also provides information on access to private health insurance, 
which, unlike information on public health insurance, is not available from administrative rec- 
ords. Where possible, the chapter presents survey estimates for parents and for three groups of 
focal children, based on the grade they were in when they entered the study: fourth grade, sev- 
enth grade, and ninth grade. 11 Most of the survey interviews were conducted between Novem- 
ber 2010 and June 2011. 12 

All survey-based measures involving health and health care are reported by the parents 
in the study sample. One potentially problematic aspect of measurement from survey-based as- 
sessment is the extent to which respondents tend to give “socially desirable” answers to ques- 
tions about their health care behavior rather than reporting on their actual behavior. This report 
is mindful of the potential problems of social desirability, as well as the accuracy of respondent 
recall, another potential problem in survey data, and, wherever possible, uses other sources (for 
instance, Seedco’s payment data or national and local studies) to try to put the Family Rewards 
survey findings in context. 

Finally, the impact analysis examines a large number of outcomes related to health care. 
As mentioned in Chapter 3, as the number of outcomes used for impact estimates increases, the 
probability of finding “false positives,” or differences that are statistically significant simply by 
chance, also increases. Although no attempt is made to formally account for the problem of 
multiple comparisons, caution must be used when interpreting impacts that do not appear to be 
part of a larger pattern of impacts within a given set of measures. 


income support through the federal Temporary Assistance for Needy Families program or New York State’s 
Safety Net Assistance program. States have some discretion to cover individuals outside those categories. 

9 Because of issues with coverage dates included in the data, Medicaid receipt in a given quarter is meas- 
ured using the recipient’s status on the first day of a quarter. As a result, the measure reported here captures 
coverage status at the start of each quarter. 

10 The survey sample for the 42 -month survey comes from the same random draw that generated the 18- 
month survey sample. However, some respondents to the 18 -month survey did not respond to the 42 -month 
survey and vice versa. Therefore, the individuals who are included in each respondent pool differ slightly. 

1 'Overall, 2,966 sample members completed the 42 -month survey interview, resulting in a response rate of 
79.3 percent. About 86 percent of the respondents completing the 42 -month survey also completed the 18- 
month survey. 

'“All interviews were completed after the end of program, and 91.6 percent of respondents were inter- 
viewed after they had received their last payment. 
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The Health Care Rewards 


Table 5.1 summarizes the Family Rewards health incentives. 13 The first two panels focus on 
rewards for preventive health and dental examinations. Each program year, families could earn 
$200 per family member for completing an annual nonemergency medical checkup. Families 
with infants and toddlers could earn another reward for having one early childhood intervention 
screening (which is free of charge to all qualifying residents of New York City), in addition to a 
regular checkup, when a physician identified potential developmental problems and recom- 
mended such an assessment. Increasing preventive health care was expected to ensure that fami- 
ly members get regular medical attention, which can lead to early diagnosis and treatment of 
health problems that can become more serious if left untreated. In addition, the program’s de- 
signers hoped that by encouraging regular care, they could increase the likelihood that partici- 
pants would establish a “medical home” — in other words, a relationship with a doctor (or 
health care institution) who knows their medical history and can provide a place for them to turn 
when problems arise, rather than resorting to hospital emergency rooms as a first response. 

Participants were expected to present doctors with an age-appropriate “preventive care 
checklist form” that identified a set of common health conditions that doctors should explore or 
screen for in any thorough annual physical examination. Separate forms were created for par- 
ents and children. The fonn was also a way to specify exactly what kind of visit to the doctor 
was to be rewarded, and it communicated to the doctor and the patient that the reward is given 
to participants for preventive behavior only — not for going to the doctor when they feel sick or 
have a particular medical problem. Specifying the activities that are supposed to take place dur- 
ing the visit is one way of distinguishing this type of doctor visit from other types of doctor vis- 
its. Figure 5.1 presents the first page of the Adult Non-Emergency Medical Checkup Fonn for 
Parents that was used in the Family Rewards program. 14 In order to claim a reward for this ac- 
tivity, parents were required to have the doctor sign this fonn at the checkup. The fonn included 
a standard set of questions to encourage the doctor to screen for common health problems. To 
encourage adherence to physician-advised follow-up care, the program also offered an incentive 
payment for one follow-up visit per family member per year over the first two years of the pro- 
gram. These rewards, as well as the above-mentioned reward for completing an early interven- 
tion assessment, were discontinued during the third year of the program because of the com- 
plexity of distinguishing visits that were truly separate follow-up visits. 


13 See Chapter 1 for further detail. 

14 The Child Non-Emergency Medical Checkup Form is shown in supplementary Appendix Figure H.1 in 
Riccio et al. (2013), which is available at www.mdrc.org. 
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The Opportunity NYC Demonstration: Family Rewards 

Table 5.1 
Health Rewards 


Activity 

Payment 

Preventive health care 

Complete annual nonemergency medical 
checkup 

$200 per family member; once a year 

Complete physician-advised follow-up 
Discontinued for Year 3 

$ 1 00 per family member; once a year 

Complete pediatrician-advised early- 
intervention referral and evaluation for 
child under 30 months 

$200 per child; once a year 

Preventive dental care 

Complete 2 dental checkups per year for 
family members age 6 and older 

$ 1 00 per family member; twice a year 

Complete 1 dental checkup per year for 
family members ages 1-5 

$100 per child; once a year 

Health insurance coverage 8 

Get or maintain health insurance coverage, 

$40 every 2 months per adult and for all 

including Medicaid, Family Health Plus, 
and/or Child Health Plus 

Discontinued for Year 3 

children combined 

Get or maintain private health insurance 

$ 1 00 every 2 months per adult and for all 

Discontinued for Year 3 

children combined 


NOTE: “Regardless of the number of children in the family, the reward payment is the 
same. For example, $100 per adult was paid every 2 months to families for getting or 
maintaining private health insurance. 


Family Rewards also offered an incentive for preventive dental care. It rewarded two 
visits per year for cleanings and checkups for all enrolled family members 6 years of age and 
older, and one per year for children between the ages of 1 and 5 years. 

The bottom panel of Table 5.1 focuses on rewards for health insurance coverage. Many 
families who enrolled in the study are eligible for means-tested public health insurance through 
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The Opportunity NYC Demonstration: Family Rewards 
Figure 5.1 

Adult Non-Emergency Medical Checkup Form for Parents (Front Page) 


OPPORTUNITY family rewards program 


ADULT Non-Emergency Medical Check-up Form 

Program Year: September 2007 -August 2008 


Patient Name: 

DOB: 

Patient #: 

Date of Visit: 






Screenings and Tests 

CONDITIONS 

GUIDELINES 

COMPLETION STATUS 

Blood pressure 

Measure at every annual visit 

C Screening completed 

Body Mass Index (BMI) 

(please refer to latest BMI chart) 

Measure at every annual visit 

□ Screening completed 

Diabetes: Fasting Plasma Glucose 

Test if: Age = 45+ and BMi 25 

L Obtained specimen for test 
No specimen obtained because: 

U Not needed for this patient or at this time 
G Patient declined 
G Other 

Smoking status (Never, past, current?) 

Ask at every annual visit 

C Screening completed 
□ If smoker, cessation recommended 

Cholesterol 

Test every 5 years for: 

• Male s age 35 

• Female > age 45 

G Obtained specimen for tost 
No specimen obtained because: 

□ Not needed for this patient or at this time 

□ Patient declined 
f) Other 

HIV status 

Ask at every annual visit and offer 
test if status is unknown 

G Obtained specimen for test 
No specimen obtained because: 

G Not needed tor this patient or at this time 
G Patient declined 
□ Other 

Depression (Use Depression Screening 
tool on back - PHQ-2) 

Ask at every annual visit 

C Screening completed 
D Patient declined 

Alcohol/Drug use (Alcohol and Drug Use 
Screening Tool on back — CAGE-AID) 

Ask at every annual visit 

G Screening completed 
C Patient declined 

Female-only. Pelvic/PapTest 

Test: • Every year until age 30 

• Every 2-3 years if > age 30 

□ Obtained specimen for test or 
referred to GYN 
No specimen obtained because: 

G Not needed for this patient or at this time 
L. Patient declined 
G Other 

Female-only. Family Planning Visit 

At every annual visit, counseling, 
referral, presenting and dispensing 
of contraceptives and screening for 
STIs 

r Counseling completed 

Female-only Mammogram 

Test every 1 -2 years if: Age > 40 

G Recommended mammogram 
G Not needed for this patient or at this time 

Colonoscopy 

For average risk, test every 10 years 
for a age 50 

C Recommended colonoscopy 

(Z Not needed for this patient or at this time 

Immunizations Review 

n Discussed immunization status and made appropriate recommendations for. 
Influenza, Pneumococcal, Hepatitis B, Tetanus and Diphtheria. 
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Medicaid, the Children’s Health Insurance Program (CHIP), 15 or Family Health Plus. 16 The cash 
incentives were intended to encourage families to keep their coverage in effect. Substantial evi- 
dence exists about the potentially harmful impacts of losing Medicaid insurance. For example, 
studies have linked loss of medical insurance with discontinuity of care, reduced ambulatory 
care use, increased emergency room use, higher health care costs, and worse patient outcomes. 17 
Furthermore, children of parents who lose Medicaid may themselves be affected through “spill- 
over effects” and are more likely to be uninsured despite their eligibility for coverage. 

Because recipients of Temporary Assistance for Needy Families (TANF) and Safety 
Net Assistance (SNA) are routinely enrolled in Medicaid and are not required to reestablish 
their eligibility as long as they remain on TANF or SNA cash welfare, 18 they were ineligible for 
this incentive while receiving those benefits, according to the program design. However, com- 
plexities in administering this portion of the Family Rewards program meant that during the 
first and second years of operation, many of these families did, in fact, receive the rewards. A 
slightly higher payment was offered to families when the parent had coverage from private, em- 
ployer-sponsored health insurance, to offset the costs of copayment that would be incurred. 

The health insurance rewards were discontinued at the start of Year 3. As noted in 
Chapter 1 , this decision was made partially because data from the early stages of the evaluation 
suggested that insurance coverage rates were very high for participants in the study and there 
was not very much room for Family Rewards to improve them. 


Receipt of Health Care Rewards 

This section briefly reviews the extent to which program members earned health care rewards 
during the three years of Family Rewards. As discussed in Chapter 2, approximately 95 percent 
of the participating families earned at least one health care reward in Year 1 of the program. 
Data for Years 2 and 3 suggest that an equally high proportion of families continued to earn 
health rewards through the second year of the program, but that this number dropped off sub- 
stantially (to approximately 73 percent) in the third and final program year. Furthermore, previ- 
ous analysis suggests that most families were fairly well informed and knowledgeable about the 


15 CHIP targets uninsured children and pregnant women in families with incomes too high to quality for 
Medicaid, but often too low for them to afford private coverage. 

16 Family Health Plus is a public health insurance program for low- income adults who are age 19 to 64 
who have income or resources too high to quality for Medicaid. Health care is provided through participating 
managed care plans in the area. 

17 See Finkelstein et al. (2012); Saunders and Alexander (2009); Fairbrother et al. (2004); Cassedy, Fair- 
brother, and Newacheck (2008); Duderstadt, Hughes, Soobader, and Newacheck (2006). 

18 Safety Net Assistance provides benefits to eligible individuals and families who do not quality for 
TANF or other federal cash assistance programs. 
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health-related activities that the program rewarded. 19 While there was some confusion about 
activities that the program did not reward, almost all survey respondents knew that they could 
earn rewards for making regular doctor or dentist visits for preventive care. This section takes a 
closer look at Seedco’s payment data to help clarify the extent to which study participants 
earned the various health care rewards that the program offered. 

Table 5.2 summarizes the rewards that parents and children in the program earned for 
preventive medical and dental visits. Data are shown for each of the three years of the program 
separately and for the three years combined. 20 Parents earned rewards for medical visits at rough- 
ly similar levels across the three years of the program. In each year, approximately 60 percent of 
parents earned at least one reward, with around 50 percent earning rewards for a doctor visit and 
about 45 percent earning rewards for a dentist visit each year. Overall, 77.9 percent of parents 
earned at least one reward during the three years of the program for either a preventive doctor or 
dental visit, with about 7 1 percent earning a reward for a doctor visit and about 65 percent earn- 
ing a reward for a dental checkup. While overall medical and dental reward receipt rates are high, 
the annual data suggest that some families earned this reward more consistently than others. 

For children, the data suggest a small drop in reward receipt over time. Around 60 per- 
cent of program households earned a reward for a child’s doctor visit and slightly more than 50 
percent earned a reward for a dentist visit each year. Those who earned at least one health care 
reward dropped somewhat from about 72 percent in Year 1 to around 65 percent in Year 3. 
Overall, 83 percent of children were in households that earned at least one preventive health 
care reward during the life of the program, with 79.6 percent earning a reward for a doctor visit 
and 72.4 percent earning a reward for a dentist visit. 

Table 5.3 shows the preventive health care rewards for children, by age and school lev- 
el. There are few differences by child’s age or school level, and high school students were not 
more likely to earn rewards than were elementary or middle school students. 21 The general pat- 
tern, however, is a small drop over time. High school students evidence larger drop-offs in re- 
ward receipt over the life of the program: 71.3 percent in Year 1 to 57.7 percent in Year 3. For 
children under the age of 30 months, reward receipt for early intervention evaluation dropped 
from 8.3 percent in Year 1 to 0.8 percent in Year, largely because children in this range at ran- 
dom assignment had aged out of Early Intervention Program eligibility after three years. 22 


19 See Riccio et al. (2010). 

20 As noted, incentives for doctor-recommended follow-up visits were discontinued in Year 3. 

21 As discussed in Chapter 1 and in Riccio et al. (2010), some education payments were made to high 
school students directly, but all other payments, including all health payments, were made to the parents. 

22 The national, state-administered Early Intervention Program provides therapeutic and support services 
for infants and toddlers with disabilities and then families. 


130 



The Opportunity NYC Demonstration: Family Rewards 

Table 5.2 

Rewards Earned for Parents’ and Children’s Doctor and Dentist Visits 


Outcome 

Year 1 Year 2 Year 3 

Years 1, 2, and 3 
Combined 

Parents (%) 





Earned at least 1 reward for medical or dental visit 

60.3 

62.4 

61.6 

77.9 

Annual physical 

49.7 

51.5 

52.0 

71.3 

Dental care checkup 

42.1 

46.0 

47.3 

64.8 

1 dental checkup 

26.4 

27.0 

27.6 

- 

2 dental checkups 

15.7 

19.0 

19.6 

- 

Doctor-recommended follow-up visit 

27.9 

32.0 

- 

44.2 

Sample size 




2,515 

Children (%) 





Earned at least 1 reward for medical or dental visit 

71.9 

68.9 

65.4 

83.0 

Annual physical 

63.0 

58.9 

59.0 

79.6 

Dental care checkup 11 

54.6 

53.1 

52.1 

72.4 

Doctor-recommended follow-up visit 

30.4 

30.3 

-- 

44.2 

Sample size 




5,680 


SOURCE: MDRC calculations using Seedco's Family Rewards program data. 

NOTES: The first program year covers September 2007 through August 2008; the second 
program year covers September 2008 through August 2009; and the third program year covers 
September 2009 through August 2010. 

"Sample size" refers to the number of individuals. 

A double dash (— ) indicates "not applicable." 

“Families can earn rewards for preventive dental visits up to twice a year for each child ages 
6 and up and once a year for children under age 6. 


Table 5.4 shows health insurance rewards earned for maintaining coverage. As this re- 
ward was discontinued in Year 3, the table reports results for Years 1 and 2 only. 23 


Impacts on Health Insurance Coverage 

As discussed in the previous report on Family Rewards and noted above, overall health insur- 
ance coverage was already extremely high for this sample when the study began. Nonetheless, 
Family Rewards helped raise it even higher. 


23 For a detailed discussion of the results for Years 1 and 2, see Chapter 7 of Riccio et al. (2010). 
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The Opportunity NYC Demonstration: Family Rewards 

Table 5.3 

Rewards Earned for Children’s Preventive Doctor and Dentist Visits, 
by Child’s Age or School Level 






Years 1, 2, and 3 

Outcome 

Year 1 

Year 2 

Year 3 

Combined 

Children under 30 months of ase at baseline (%) 





Child earned at least 1 reward for medical or dental visit 

76.4 

73.6 

68.5 

87.0 

Annual physical 

70.1 

67.7 

63.0 

85.8 

Early intervention evaluation 

8.3 

1.6 

0.8 

10.6 

Dental care visit a 

27.6 

43.7 

48.8 

60.6 

Doctor-recommended follow-up visit 

34.7 

33.5 

- 

50.0 

Sample size 




254 

Children 30 months to 5 years of ase at baseline (%) 

Child earned at least 1 reward for medical or dental visit 

73.5 

74.0 

72.0 

85.6 

Annual physical 

67.1 

64.7 

66.5 

83.9 

Dental care visit a 

54.8 

59.6 

57.2 

75.7 

Doctor-recommended follow-up visit 

33.6 

35.7 

- 

49.7 

Sample size 




547 

Elementary school students at baseline 1%) 





Child earned at least 1 reward for medical or dental visit 

74.3 

73.9 

71.7 

85.6 

Annual physical 

65.9 

63.5 

65.3 

83.0 

Dental care visit a 

59.6 

60.1 

59.0 

77.2 

Doctor-recommended follow-up visit 

31.2 

33.8 

- 

46.3 

Sample size 




1,889 

Middle school students at baseline 1%) 





Child earned at least 1 reward for medical or dental visit 

72.4 

70.7 

68.8 

85.4 

Annual physical 

63.5 

60.2 

62.8 

82.3 

Dental care visit a 

56.1 

53.9 

55.3 

73.8 

Doctor-recommended follow-up visit 

31.5 

30.6 

- 

45.1 

Sample size 




1,264 

Hish school students at baseline 1%) 





Child earned at least 1 reward for medical or dental visit 

71.3 

63.9 

57.7 

80.7 

Annual physical 

61.0 

52.9 

50.9 

75.4 

Dental care visit a 

54.9 

48.4 

45.1 

70.6 

Doctor-recommended follow-up visit 

28.5 

26.7 

- 

41.3 

Sample size 




1,538 


(continued) 
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Table 5.3 (continued) 


SOURCE: MDRC calculations using Seedco's Family Rewards program data. 

NOTES: The first program year covers September 2007 through August 2008; the second program 
year covers September 2008 through August 2009; and the third program year covers September 
2009 through August 2010. 

"Sample size" refers to the number of children. 

A double dash (— ) indicates "not applicable." 

“Families can earn rewards for preventive dental visits up to twice a year for each child ages 6 
and up and once a year for children under age 6. 


The Opportunity NYC Demonstration: Family Rewards 

Table 5.4 

Health Insurance Rewards Earned for Maintaining Coverage 


Outcome 

Year 1 

Year 2 Year3 a 

Years 1, 2, and 3 
Combined 

Parents (%) 

Earned at least 1 reward for 
Parent's public heath insurance 
Parent's private health insurance 
Parent's public or private health insurance 

Sample size 

66.5 

20.1 

82.3 

68.3 

20.0 

84.5 

74.3 
23.7 

90.4 

2,515 

Families (%) 




Earned at least 1 reward for 




Children's public health insurance 

70.8 

71.3 

77.2 

Children's private health insurance 

15.4 

13.9 

17.8 

Children's public or private health insurance 

81.5 

81.6 

88.1 

Sample size 



2,377 


SOURCE: MDRC calculations using Seedco's Family Rewards program data. 

NOTE: The first program year covers September 2007 through August 2008; the second 
program year covers September 2008 through August 2009; and the third program year 
covers September 2009 through August 2010. 

“These rewards were discontinued in Year 3. 
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Health Insurance Coverage Overall 


The analysis of health insurance coverage draws on both the 42-month survey and ad- 
ministrative records data. The survey provides a more comprehensive snapshot of health insur- 
ance coverage by including a series of questions on whether participants — and members of 
their families — were covered by public or private health insurance in the month before the sur- 
vey interview. The administrative records data, on the other hand, capture receipt of public in- 
surance only, but offer multiple data points, pre- and post-random assignment, allowing the 
study to trace impacts on Medicaid and other public health insurance receipt over time. There- 
fore, these two data sources offer slightly different lenses for assessing the program’s effects on 
health insurance coverage. 

Table 5.5 presents impacts on families’ health insurance coverage. Starting with the 
survey-based results, it is evident that at about 42 months after random assignment (the average 
length of the survey follow-up period), few sample members were without health insurance 
coverage. The control group coverage rates are higher than the designers of Family Rewards 
anticipated, especially for people living in the very high-poverty neighborhoods from which the 
sample was selected. A number of factors may be contributing to the higher rates found in the 
Family Rewards sample. First, the study sample does not include undocumented immigrants, 
who tend to have much lower coverage rates than legal immigrant and nonimmigrant low- 
income families. Nor does it include singles without children, another group with historically 
low coverage rates. Second, as discussed previously, New York City and New York State have 
expanded public health insurance coverage for low-income families (Medicaid program, Family 
Health Plus, and CHIP) and have intensified their efforts to market that coverage to those fami- 
lies in recent years. Along with the efforts to expand coverage, the city and state have also made 
significant investments in strengthening the primary and preventive health care delivery ys- 
tem. 24 Moreover, many states “chum” Medicaid recipients at a higher rate, requiring them to 
renew coverage every six months, compared with every year in New York. In an attempt to re- 
duce the number of parents and children churning on and off Medicaid and CHIP, New York 
has implemented several steps to streamline and simplify Medicaid renewal processes. 25 Never- 
theless, program group respondents were more likely (95.5 percent versus 92.9 percent, a dif- 
ference of 2.7 percentage points) to report having health insurance in the previous month. 26 


24 Coalition of New Y ork State Public Health Plans (2009). 

25 United Hospital Fund (2009). 

26 More program group respondents than control group respondents reported that their children had health 
insurance coverage, though this comparison is nonexperimental and no statistical testing was done for it. About 
80 families either refused to discuss their children or noted that they had a child age 19 or older who no longer 
lived with them (for whom no coverage information was obtained); they are not included in the calculation. 
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Table 5.5 


Impacts on Families’ Health Insurance Coverage 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Health insurance coverage in Drevious month t%l 

Respondent had health insurance 

95.5 

92.9 

2 7 *** 

0.002 

Publicly funded 

71.9 

71.0 

0.9 

0.545 

Privately funded 

32.5 

28.1 

4 4 *** 

0.003 

Publicly and privately funded 

8.9 

6.1 

2 7 *** 

0.005 

All dependent children had health insurance a 

95.0 

92.6 

- 

- 

Public health insurance only h 

74.8 

74.9 

- 

- 

Private health insurance only b 

17.5 

16.0 

- 

- 

Health insurance coverage in Dast 12 months (%) 

Respondent had a period with no coverage 

15.3 

17.6 

-2.3 * 

0.087 

Some or all of respondent's children had a 

period with no coverage a 

13.9 

17.2 

- 

- 

Sample size (total = 2,966) 

1,543 

1,423 




SOURCE: MDRC calculations using data from the Family Rewards 42 -month survey. 


NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the differences between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Italic type indicates comparisons that are nonexperimental. Statistical tests were not performed. 

“Child-related health insurance measures were calculated for sample members with at least 1 child 
living with them at the time of the survey. 

b The percentages of all children covered by public health insurance and all children covered by 
private health insurance do not add up to the percentage of all children covered by any insurance 
because some families reported having children covered by both types of insurance. 


Continuous Health Insurance Coverage 

Low-income families experience a high rate of interruptions — or churning — in 
public health coverage. Despite recent efforts to streamline and simplify the Medicaid renew- 
al process, it is estimated that approximately one-third of New York’s public health enrollees 
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fail to complete the recertification process each year and lose coverage as a result. 27 Even 
though they remain eligible, some families lose coverage at the time of recertification because 
they fail to successfully complete the process. Furthermore, gaps in coverage also occur with 
private insurance programs. 

Program group respondents (see Table 5.5) were somewhat less likely than control 
group respondents to report a period of no insurance coverage since random assignment (15.3 
percent versus 17.6 percent, a decrease of 2.3 percentage points). 

Public Health Insurance Coverage 

On the 42-month parent survey, program and control groups report fairly comparable 
levels of public health insurance coverage: 71.9 percent of the program group and 71.0 percent 
of the control group reported having such insurance at the time of the interview, a difference of 
0.9 percentage point that is not statistically significant, as shown in Table 5.5. The public health 
insurance levels reported at the point of the survey interview are fairly comparable with the lev- 
els reported upon entry into the study. 

Appendix Table D.l displays impacts on parents’ Medicaid coverage. It lists Medicaid 
coverage for each of the three program years and for the three years combined. About 80 per- 
cent of both control and program group parents were ever covered by Medicaid during the three 
years of the program. Parents in the program group were more likely than control group parents 
to have had continuous coverage during the program period (37.3 percent versus 34.8 percent, a 
difference of 2.5 percentage points). This result is statistically significant for Year 1 (2.6 per- 
centage points) and Year 2 (2.9 percentage points). There are, however, no statistically signifi- 
cant differences in Year 3. 

Appendix Table D.2 displays impacts on parents’ Medicaid coverage by receipt of 
TANF or SNA status at the time of random assignment. Because individuals receiving TANF 
or SNA benefits are automatically enrolled in Medicaid, they did not receive rewards during the 
first two years of the program for maintaining insurance coverage. 28 Therefore, it was less likely 
that program participation would produce impacts on coverage for the subgroup receiving 
TANF or SNA benefits at the time of random assignment. 


27 Lake Research Partners and Perry (2009). 

28 In Year 1, many recipients of TANF or SNA did receive rewards for maintaining public health insur- 
ance, even though they were not eligible for those rewards. Although Seedco staff corrected for the overpay- 
ments when they discovered the error by debiting the overpaid amounts from future reward payments, they 
were unable to make the correction in all cases. 
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Among those who were not receiving TANF or SNA benefits at the time of random as- 
signment, there is a difference between program and control groups in both the percentage who 
were always covered by Medicaid (that is, without interruption) and the average number of 
quarters covered. For example, 34.5 percent of the program group always had coverage, com- 
pared with 31.1 percent of the control group — a statistically significant difference of 3.4 per- 
centage points. The effect persists for each individual year. Overall, there is no difference in 
coverage between the program and control groups among parents who were receiving TANF or 
SNA benefits at the time of random assignment. 

There are no differences between the program and control groups in children’s Medic- 
aid coverage for the sample overall, as shown in the supplement to this report. 29 


Impacts on Receipt of Health Care Services 

Parents’ Receipt of Health Care Services 

As discussed previously, Family Rewards incentives are also intended to promote regu- 
lar preventive medical and dental care for all family members. Adopting good preventive health 
care practices is linked to healthier lives by reducing unmet health care needs. National data 
from one study show that 81.8 percent of adults 18 years of age and over had contacted a doctor 
or other health professional within the previous 12 months (excluding overnight hospital 
stays). 30 While it is unclear whether these visits were for regular checkups, these rates are lower 
than the rates of preventive health care visits reported by the low-income Family Rewards sam- 
ple. The program’s designers expected that the opposite would be the case. 

The 42-month survey attempts to capture receipt of preventive health care services by 
asking respondents whether they had gone to a doctor, hospital, or clinic for a routine physical 
checkup (when they were not sick) at any time after random assignment. A separate question 
asked whether they had seen their personal doctor or health care provider for a checkup or any 
medical care in the prior 12 months. Similarly, since the program encourages participants to 
have two dental checkups a year, survey respondents were asked how many times they had seen 
a dentist for a routine checkup or to have their teeth cleaned. 

Findings for parents’ receipt of preventive health care services are shown in Table 5.6. 
The data show no effect on health checkups, visits to emergency rooms, or overnight hospital 


29 See supplementary Appendix Table H.7 in Riccio et al. (2013), available at www.mdrc.org. 

30 See Schiller, Lucas, and Peregoy (2012). Another study, using a 19-month window, finds that 90 percent 
of all Americans, 74 percent of African-Americans, and 70 percent of children age 5 to 15 received preventive 
exams in that period; see Cherry, Woodwell, and Rechtsteiner (2007). 
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admissions. Program group parents were, however, more likely than those in the control group 
to report having had at least one dental checkup (81.3 percent versus 71.4 percent, a difference 
of 10 percentage points) or two or more dental checkups (45.2 percent versus 33.5 percent, 
a difference of 1 1 .8 percentage points) in the previous 12 months. 

Source of Health Care 

The 42-month survey included a question that asked respondents whether they had a 
regular source of care — a place where they usually go when they are sick or in need of ad- 
vice about their health, or when they need routine health care, like a checkup. Those who 
gave a positive response to the question were asked whether they go to a clinic or health 
center, a doctor’s office or HMO, a hospital emergency room, a hospital outpatient depart- 
ment, or some other place for care. 31 This question does not attempt to measure actual use 
of the emergency room or hospital outpatient department; rather, it attempts to assess 
whether families have a “usual” place of care to which they turn when needed for routine 
health care purposes. 

By encouraging participants to have a regular health care provider, the program 
promoted the “medical home” concept, a model of health care delivery that, as explained 
earlier, includes an ongoing relationship between provider and patient, and a comprehensive 
approach to care and coordination of care through providers and community services. 32 
Such a model of health care delivery is expected to reduce reliance on hospital emergency 
rooms for routine care. Emergency room costs per visit are generally much higher than the 
costs of delivering comparable care in an outpatient clinic or doctor’s office. At 42 months, 
as shown in Table 5.6, parents in the program group were more likely than those in the con- 
trol group to report using a clinic or health center as a usual source of health care (59.5 per- 
cent versus 52.7 percent, a difference of 6.8 percentage points) and less likely to report hav- 
ing a doctor’s office as a usual source of health care (21.8 percent versus 25.1 percent, a 
reduction of 3.3 percentage points). There was no effect on use of hospital emergency 
rooms as a usual source of care. 


3 'National estimates for this type of measure suggest that 16 percent of adults who are 18 years of age and 
over were without a usual place of health care. Of those with a usual place of care, 77 percent considered a 
doctor’s office or HMO to be their usual place of health care, 20 percent considered a clinic or health center to 
be their usual place of health care, and 3 percent considered a hospital emergency room or outpatient depart- 
ment to be their usual place of health care; see Pleis, Lucas, and Ward (2009). 

3 ~The concept of the medical home has evolved since its introduction by the American Academy of Pedi- 
atrics in 1967. Research suggests that individuals who have continuity with a regular practitioner are more like- 
ly to adhere to prescribed medications and to receive preventive care and well-coordinated, resource-efficient, 
and family-centered care, and are less likely to visit the emergency department and be hospitalized; in addition, 
their practitioner is more likely to recognize their problems and track their information. See Christakis et al. 
(2001, 2002, 2003) and Starfield and Shi (2004). 


138 



The Opportunity NYC Demonstration: Family Rewards 

Table 5.6 

Impacts on Parents’ Receipt of Health Care Services 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

ResDondent's health care use 1%) 

Has a usual source of health care 

94.6 

93.5 

1.2 

0.277 

Clinic or health center 

59.5 

52.7 

6.8 *** 

0.002 

Doctor's office 

21.8 

25.1 

-3.3 * 

0.067 

Hospital emergency room 

3.0 

4.0 

-1.0 

0.238 

Hospital outpatient department 

9.9 

11.3 

-1.3 

0.331 

Other 

0.5 

0.4 

0.0 

0.924 

Has seen health professional for any reason 

in past 12 months 

94.4 

94.5 

-0.1 

0.924 

Had a health checkup in past 12 months 

90.0 

88.9 

1.1 

0.427 

Number of visits to dentist for any reason 
in past 12 months 

None 

14.6 

24.8 

-10 1 *** 

0.000 

1 

22.0 

25.2 

-3.2 * 

0.100 

2 

38.6 

28.1 

10.5 *** 

0.000 

3 or more 

24.7 

21.9 

2.8 

0.149 

Has seen dentist for any reason in past 

12 months 

85.4 

75.3 

|Q | *** 

0.000 

Had 1 or more dental checkups in past 

12 months 

81.3 

71.4 

10.0 *** 

0.000 

Had 2 or more dental checkups 

45.2 

33.5 

1 1 g *** 

0.000 

Number of visits to emergency room 
in past 12 months 

None 

55.8 

53.8 

2.0 

0.378 

1 

17.2 

19.6 

-2.4 

0.177 

2 

13.6 

14.6 

-1.0 

0.512 

3 or more 

13.4 

12.0 

1.5 

0.333 

Number of overnight admissions to hospital 
in past 12 months 

None 

86.1 

87.3 

-1.2 

0.426 

1 

7.6 

7.3 

0.3 

0.777 

2 or more 

6.3 

5.4 

0.9 

0.404 


(continued) 
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Table 5.6 (continued) 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Unmet health needs due to cost in Dast 
12 months (%) 





Did not get needed medical care 11 

7.1 

8.1 

-1.1 

0.371 

Did not fill prescription 

14.4 

13.0 

1.3 

0.393 

Sample size (total = 1,961) 

1,022 

939 




SOURCE: MDRC calculations using data from the Family Rewards 42 -month survey. 

NOTES: The items in this section of the survey were administered to a random subsample (N = 
1,961) of the survey respondents. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the difference between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; 
** = 5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 
a This excludes prescriptions. 


Unmet Health Needs 

In order to determine the program’s effects on reducing unmet health needs, the 42- 
month survey included questions that asked respondents whether they needed to see a doctor or 
fill a prescription for medicine in the prior year but had not done so because of the cost. At 42 
months after random assignment, there were no differences between the program and control 
groups in reported unmet health care needs because of costs. 

For nondental health care use, overall percentages in both the control and program 
groups are similar to those found in the 18-month survey. 33 On the other hand, effects on hav- 
ing a usual source of health care or unmet health needs that were detected in the 18-month 
survey had faded by the time of the 42 -month survey, except for having a clinic or health cen- 
ter as the usual source of health care. The number of dental checkups fell in both groups be- 
tween the two surveys, 34 though the drop-off was steeper for control group members, produc- 

33 See Riccio et al. (20 10). While there is significant overlap in respondents between the 18-month and 42- 
month surveys, the latter included some individuals who were not in the original sample. This does not change 
the interpretation of the findings. 

34 The 18-month survey asked solely about dental checkups, but the numbers suggest that respondents may 
have included any other dental visits when answering this question. As a result, in the 42 -month survey, 

(continued) 
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ing a statistically stronger and quantitatively larger difference in dental care use between the 
program and control groups. 35 

Children’s Receipt of Health Care Services 

Findings for focal children’s receipt of preventive health care services are shown in Ta- 
ble 5.7. (Focal children continue to be counted in their original grade category in Year 3 even if 
they have aged into the next school-level category.) 36 

Children under the age of 5 years at the time of random assignment. There are no 
statistically significant differences between the control and program groups in routine checkups, 
dental checkups, or Early Intervention Program screening for children under the age of 5. 37 

Elementary school students at the time of random assignment. For focal children 
who were in elementary school at the time of random assignment, there are almost no statistical- 
ly significant differences in nondental health care use. Effects on receipt of dental care are 
strong. Program group children are more likely to have had one or more dental checkups (94.1 
percent versus 89.6 percent, a difference of 4.5 percentage points) and two or more dental 
checkups (61.5 percent versus 46.4 percent, a difference of 15.0 percentage points) in the previ- 
ous 12 -month period. 

Middle school students at the time of random assignment. Program group children 
were more likely to have had a health checkup or received shots in the past 12 months than 
were those in the control group (97.4 percent versus 93.6 percent, a difference of 3.8 percentage 
points). They were also more likely to have had one or more dental checkups (94.2 percent 


MDRC asked about routine (preventive) dental checkups and other dental visits separately. While this change 
brought responses more in line with expectations, a significant minority of respondents continued to answer the 
question on checkups in a way that suggests they included dental visits for other reasons. 

35 Though Medicaid covers dental expenses, only about 20 percent of dentists accept it (U.S. General Ac- 
counting Office, 2000). This can make it difficult to find a provider who accepts Medicaid. Low-income 
households may, therefore, have to pay for dental care out-of-pocket. Dental care (especially a second visit 
each year) may be seen as a lower priority and thus an obvious place to cut back during financial hardship or in 
anticipation of future hardship such as during the deep economic recession that occurred during the program 
period. It appears that program participation is associated with sustained or at least less attenuated use of rou- 
tine dental care. The cash rewards (still available to most respondents during the year before the survey) could 
have provided a means to sustain dental care that participants would otherwise not have had. 

36 For example, children who were in elementary school (grade 4) at baseline are counted in this category 
even though they had aged into the middle school category (grade 7) by the time of the 42 -month survey. 

37 In the 18-month survey, this category included children up to age 6. 
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Table 5.7 

Impacts on Focal Child’s Receipt of Health Care Services, 
by Child’s Age or School Level 



Program 

Control 

Difference 


Outcome 

Group 

Group 

(Impact) 

P-Value 

Children under ase 5 at baseline 3 1%) 

Had routine health checkup in past year b 

97.9 

97.2 

0.7 

0.650 

Had dental checkup in past year’ 1 c 

59.2 

57.0 

2.2 

0.690 

Any children screened for Early Intervention 
Program 1 ' 

30.4 

25.6 

4.8 

0.262 

Sample size (total = 456) 

239 

217 



Elementary school students at baseline 1%) 





Child has usual source of routine care 

93.6 

90.9 

2.7 

0.132 

Child has personal pediatrician 

89.4 

89.0 

0.4 

0.861 

Child has more than 1 personal pediatrician 

12.1 

9.2 

2.8 

0.162 

Child has seen health professional for any 
reason in past 12 months 

98.4 

98.0 

0.4 

0.677 

Child has seen personal pediatrician for any 
reason in past 12 months 
Child had health checkup or got shots in past 

78.3 

77.8 

0.5 

0.855 

12 months 

97.3 

96.7 

0.5 

0.648 

Child has usual source of care when sick 

93.8 

94.1 

-0.3 

0.842 

Clinic or health center 

59.8 

59.5 

0.3 

0.919 

Doctor's office 

26.0 

25.5 

0.5 

0.870 

Hospital outpatient department 

5.7 

6.3 

-0.7 

0.663 

Hospital emergency room 

1.7 

2.8 

-1.1 

0.279 

Other place 

0.6 

0.0 

0.7 * 

0.086 

Child saw dentist for any reason in past 12 months 

94.2 

90.1 

4 [ ** 

0.024 

Child had 1 or more dental checkups in past 12 
months 

94.1 

89.6 

4.5 ** 

0.013 

2 or more dental checkups 

61.5 

46.4 

15.0 *** 

0.000 

Sample size (total = 939) 

486 

453 




(continued) 
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Table 5.7 (continued) 



Program 

Control 

Difference 


Outcome 

Group 

Group 

(Impact) 

P- Value 

Middle school students at baseline (%) 





Child has usual source of routine care 

93.2 

90.7 

2.5 

0.174 

Child has personal pediatrician 

89.5 

90.0 

-0.5 

0.815 

Child has more than 1 personal pediatrician 

12.5 

13.5 

-1.0 

0.648 

Child has seen health professional for any reason 
in past 12 months 

98.9 

95.9 

2 0 *H=H= 

0.004 

Child has seen personal pediatrician for any 





reason in past 12 months 

79.6 

77.4 

2.3 

0.412 

Child had health checkup or got shots in past 

97.4 

93.6 

3 g H=H=H= 

0.006 

12 months 





Child has usual source of care when sick 

95.5 

93.4 

2.1 

0.158 

Clinic or health center 

65.6 

60.5 

5.1 

0.113 

Doctor's office 

23.4 

24.8 

-1.4 

0.628 

Hospital outpatient department 

5.2 

6.7 

-1.5 

0.342 

Hospital emergency room 

1.4 

1.3 

0.1 

0.943 

Other place 

0.0 

0.2 

-0.1 

0.511 

Child saw dentist for any reason in past 12 months 

94.4 

86.8 

"j y *** 

0.000 

Child had 1 or more dental checkups in past 
12 months 

94.2 

86.5 

y y *** 

0.000 

2 or more dental checkups 

61.6 

46.9 

14 g *** 

0.000 

Sample size (total = 905) 

473 

432 



High school students at baseline (%) 





Child has usual source of routine care 

93.1 

90.5 

2.6 

0.183 

Child has personal pediatrician 

88.0 

85.9 

2.1 

0.369 

Child has more than 1 personal pediatrician 

14.9 

9.8 

5.1 ** 

0.031 

Child has seen health professional for any reason 
in past 12 months 

96.4 

96.4 

0.1 

0.953 

Child has seen personal pediatrician for any 
reason in past 12 months 
Child had health checkup or got shots in past 

78.0 

73.2 

4.8 

0.115 

12 months 

94.0 

94.3 

-0.3 

0.876 

Child has usual source of care when sick 

93.6 

93.3 

0.4 

0.832 

Clinic or health center 

55.6 

56.2 

-0.6 

0.870 

Doctor's office 

26.4 

25.6 

0.8 

0.785 

Hospital outpatient department 

8.5 

7.5 

1.0 

0.598 

Hospital emergency room 

2.5 

3.2 

-0.8 

0.522 

Other place 

0.7 

0.8 

-0.1 

0.822 
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Table 5.7 (continued) 



Program 

Control 

Difference 


Outcome 

Group 

Group 

(Impact) 

P-Value 

Child saw dentist for any reason in past 12 months 

93.8 

89.1 

4 7 ** 

0.021 

Child had 1 or more dental checkups in past 
12 months 

93.0 

88.6 

4 4 ** 

0.037 

2 or more dental checkups 

62.9 

44.1 

18.8 *** 

0.000 

Sample size (total = 812) 

429 

383 




SOURCE: MDRC calculations using data from the Family Rewards 42 -month survey. 

NOTES: This table presents outcomes for the randomly selected focal child only, who must have 
been living in the household at the time of interview. 

Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the differences between outcomes for the program 
and control groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 
percent; ** = 5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

“In the previous Family Rewards survey (the 18 -month survey), this section was asked of 
respondents with a child under age 6 in the household; however, in the 42-month survey, this 
section was asked of respondents with a child under age 5 in the household. 

b This measure pertains only to a child in the household under age 5 who was randomly selected 
for the respondent to discuss. 

c If the randomly selected child is less than 1 year old, then the child is excluded from this 
measure. 

d This includes any of the respondent's children under age 5. 


versus 86.5 percent, a difference of 7.7 percentage points) and two or more dental checkups 
(61.6 percent versus 46.9 percent, a difference of 14.8 percentage points) in the previous 12- 
month period. 

High school students at the time of random assignment. There were almost no 
differences between high school-age children in the program and control groups in nondental 
health care use. Program group high school-age children were, however, more likely to have had 
one or more dental checkups (93.0 percent versus 88.6 percent, a difference of 4.4 percentage 
points) and two or more dental checkups (62.9 percent versus 44.1 percent, a difference of 18.8 
percentage points) than those in the control group. With the exception of changes in the 
frequency of dental checkups, numbers for children’s receipt of health care services remain 
consistent in magnitude across surveys for both the control and program groups. In some cases 
weaker statistical effects (10 percent level) disappeared or emerged between surveys. On the 
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other hand, effects on the frequency of dental checkups emerged and became stronger. As of the 
42 -month survey, there were large and strongly statistically significant effects on dental care for 
elementary school students and middle school students. The effect for high school students at 42 
months extends to the receipt of one or more checkups, and the impact size on having received 
two or more checkups increases to 18.8 percentage points. As with parents’ results, the effect 
appears to emerge from a drop in the frequency of dental visits for the control group. Program 
group respondents maintain previous levels of dental care receipt or else drop care at reduced 
rates. Again, as already noted, it is possible that incentive payments helped program participants 
continue to receive dental care (often effectively uncovered given the small number of dentists 
who accept Medicaid and possibly considered a domain for reduced spending) during the 
significant economic downturn that began in 2008. 


Impacts on Health Outcomes 

If Family Rewards encourages continuous health care coverage and preventive health care vis- 
its, a logical question is whether this behavior translates into improved health outcomes for both 
parents and children. In the short run, Family Rewards is expected to improve health status 
through more diagnosis and management of chronic health conditions. Through its focus on 
health, the program might also have potential “spillover” effects on other health risk-taking be- 
havior, such as smoking and lifestyle habits that lead to obesity, both of which are linked to 
mortality and a wide range of health problems and costs. 38 By conditioning reward payments on 
preventive health care activities, and by providing participants with the health care checklist to 
guide the rewarded interactions with doctors, it is possible that the program can generate sec- 
ondary benefits, such as reductions in the types of behavior that have documented health risks. 

Parents’ Health Outcomes 

Self-Reported Health Status 

Table 5.8 presents several measures for assessing the program’s effects on health out- 
comes. The first item is a widely used and validated global self-assessment of health status. Per- 
ceived health status is assessed based on an item drawn from the “Short Form 12 Health Sur- 


38 0besity has been linked to an increased risk of numerous comorbidities, including high blood pres- 
sure, high blood cholesterol, type 2 diabetes mellitus, coronary heart disease, osteoarthritis, asthma, and 
gallbladder disease; see Must et al. (1999) and Mokdad et al. (2003). Moreover, obesity has been found to 
significantly lower life expectancy, particularly among young adults; see Fontaine et al. (2003). With the 
rise in obesity, poor diet and physical inactivity have now become the number two preventable causes of 
death in the United States, behind only tobacco in the number of lives claimed each year; see Mokdad, 
Marks, Stroup, and Gerberding (2004). 
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vey” (SF12), a widely used scale in national health surveys that provides a generic, multi- 
dimensional measure of physical and mental health. 39 The Family Rewards 42 -month survey 
respondents were asked, “Would you say your health in general was excellent, very good, good, 
fair, or poor?” When the same question was posed to a national sample, 13 percent of U.S. 
adults who were 1 8 years of age and over assessed themselves as being in fair or poor health. 
Adults in near-poor and poor families were two to three times more likely to have fair or poor 
health compared with adults in families who were not poor. 40 About one-third of parents in the 
Family Rewards study gave such a response, and, overall, the program produced no differences 
in perceived health status at 42 months after random assignment. 

Health Conditions 

Survey respondents were also asked whether they had a medical or health condition, 
such as high blood pressure, high cholesterol, asthma, cancer, diabetes, or another health prob- 
lem. They were asked to indicate whether the conditions that they mentioned were being treat- 
ed. Overall, the program had no impact on the likelihood of reporting a medical condition, 41 or 
of receiving treatment for any medical condition. 

Health Risks 

Table 5.8 also presents findings for selected behaviors that are known to have serious 
health implications: smoking and lifestyle habits that lead to being overweight. To explore the 
potential impacts of the program, this section presents the estimated effects of Family Rewards 
on smoking and obesity. 

Nationally, the prevalence rate for smoking by individuals who are 18 years of age or 
older in 2010 was estimated to be 19.3 percent. 42 The Family Rewards control group reports a 
somewhat higher rate than that (24.5 percent). Survey respondents were asked whether they 
were currently smoking and the number of cigarettes smoked a day. There were no statistically 
significant differences between the program and control groups on this outcome. 

39 See Hays, Sherbourne, and Mazel (1995). 

40 This is consistent with the National Health and Nutrition Examination Survey (NHANES) finding that 
Food Stamp program (FSP) participants have a more negative perception of their health status than do higher- 
income nonparticipants. About a third of the FSP participants rated their health status as very good or excellent, 
and a third rated their health status as fair or poor. The NHANES also finds that physician assessments of gen- 
eral health status are consistently more positive than are self-assessments. See www.cdc.gov/nchs/nhanes.htm. 

4 1 T able 5.8 shows only the four most common health conditions that respondents mentioned. 

'“The data suggest that adults with low educational attainment had the highest prevalence of smoking — 
that is, 45.2 percent among persons with a General Educational Development (GED) certificate, compared 
with 6.3 percent among those with a graduate degree. The prevalence of current smoking is also higher among 
adults who live below the federal poverty level (28.9 percent). See Centers for Disease Control and Prevention 
( 2011 ). 
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Table 5.8 


Impacts on Parents’ Health Outcomes 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Effect 

Size 

Health status 

Average self-rated health 

(1 = poor; 5 = excellent) 

3.1 

3.1 

0.1 

0.165 

0.056 

Excellent (%) 

14.6 

14.1 

0.5 

0.759 


Very good (%) 

21.3 

18.8 

2.5 

0.149 


Good (%) 

33.4 

34.3 

-0.9 

0.667 


Fair (%) 

24.3 

25.6 

-1.3 

0.500 


Poor (%) 

6.4 

7.2 

-0.8 

0.452 


Currently pregnant (%) 

1.3 

2.4 

-1.1 * 

0.083 


Health conditions and risks 

Smokes cigarettes (%) 

22.6 

24.5 

-1.9 

0.314 


Has any medical condition 11 (%) 

55.4 

53.8 

1.5 

0.463 


Asthma 

17.3 

16.3 

1.0 

0.561 


High blood pressure/hypertension 

27.4 

26.7 

0.8 

0.689 


High cholesterol/high LDL 

10.1 

11.4 

-1.3 

0.326 


Diabetes 

11.5 

11.1 

0.4 

0.759 


Treated for any medical condition' 1 (%) 

50.1 

47.8 

2.3 

0.276 


Asthma 

15.0 

14.9 

0.1 

0.942 


High blood pressure/hypertension 

25.6 

24.8 

0.8 

0.672 


High cholesterol/high LDL 

9.0 

9.4 

-0.5 

0.709 


Diabetes 

11.2 

10.4 

0.8 

0.568 


Average Body Mass Index (BMI) b 

30.6 

30.6 

0.0 

0.905 

-0.006 

Underweight (%) 

1.0 

0.6 

0.5 

0.258 


Normal weight (%) 

20.2 

21.7 

-1.6 

0.407 


Overweight (%) 

32.8 

31.3 

1.5 

0.491 


Obese (%) 

46.0 

46.4 

-0.4 

0.864 


Psychosocial well-being 

Average score on "State of Hope" scale 

(6 = low; 24 = high) 1 

17.9 

17.5 

0 2 *** 

0.005 

0.127 

How life today compares to way it was 
a year ago (%) 

Much better today 

32.5 

31.7 

0.8 

0.715 


Somewhat better 

26.7 

27.7 

-1.0 

0.630 


About the same 

28.1 

27.3 

0.8 

0.677 


Somewhat worse 

7.8 

7.6 

0.2 

0.869 


Much worse 

4.9 

5.7 

-0.9 

0.403 


Sample size (total = 1,961) 

1,022 

939 
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Table 5.8 (continued) 

SOURCE: MDRC calculations using data from the Family Rewards 42 -month survey. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and 
control groups. The p-value indicates the likelihood that the difference between the program 
and control groups arose by chance. Statistical significance levels are indicated as follows: *** 

= 1 percent; ** = 5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre- 
random assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

The effect size is the difference between program and control group outcomes expressed as 
a proportion of the standard deviation of the outcomes for both groups combined. 

A randomly selected subsample of survey respondents (N = 1,961) was asked these 
questions. 

“The 4 most commonly reported conditions are listed. 

b Weight categories are from the National Institutes of Health. Underweight is defined as 
having a BMI of less than 18.5. Normal weight is defined as having a BMI between 18.5 and 
24.9. Overweight is defined as having a BMI between 25.0 and 29.9. Obesity is defined as 
having a BMI of at least 30. About 6 percent of the sample is excluded from this analysis 
because of missing data. 

c The "State of Hope" scale measures the level of ongoing goal-directed thinking. The 
response codes (1 to 4) of the 6 items for each person are summed, with lower values 
representing less goal-directed thinking and higher values representing more. The scale is taken 
from Snyder et al. (1996). 

Obesity is another pressing health concern, with many experts claiming that the country 
is suffering from an obesity epidemic. The concerns about the increasing prevalence of obesity 
are founded in the association between obesity and adverse health outcomes, including excess 
mortality, and increased health expenditures. 43 The measure used most often to assess this con- 
dition is referred to as the Body Mass Index, or BMI. It is calculated from the sample members’ 
responses to survey questions regarding height and weight. 44 As shown in Table 5.8, there are 
no statistically significant differences in the BMI scores for the Family Rewards program and 
control groups. However, around 46 percent of the sample members in each group were classi- 
fied in the obese category. 


43 Schmeiser (2008); Flegal, Graubard, Williamson, and Gail (2005). 

44 “Underweight” is defined as a BMI of less than 18.5, “normal weight” as greater than or equal to 18.5 
and less than 25, “overweight” as greater than or equal to 25 and less than 30, and “obese” as greater than or 
equal to 30. Using self-reported measures of height and weight on the 42 -month survey, BMI scores were cal- 
culated for the program and control group members in the Family Rewards sample. 
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Psychosocial Well-being 

The final set of measures that is shown in the table captures two dimensions of psy- 
chological well-being: psychological distress and the belief in one’s ability to initiate and sus- 
tain action. 45 Consistent with prior research, it is possible that changes in health and economic 
well-being brought about by such programs could improve mental health and emotional well- 
being. The Family Rewards findings offer a small bit of support for this hypothesis. As shown 
in Table 5.8, program group respondents, on average, score slightly higher on the State of 
Hope scale used to measure psychosocial well-being (17.9 units versus 17.5 units, a 0.3 unit 
increase on a scale from 6 to 24 and a statistically significant increase of 1.7 percent over the 
control group value). 

In summary, the magnitudes for measures of health status, medical conditions, and psy- 
chosocial well-being in both the control and program groups are consistent with those reported 
in the 18-month survey. The earlier report on Family Rewards found statistically significant 
effects on average self-rated health. Program group respondents were also more likely to report 
excellent health and less likely to report poor health. 46 These effects were no longer evident by 
the time of the 42-month survey, suggesting that the initial effects had dissipated or previously 
were the product of statistical “noise.” The effect on the State of Hope scale reported above was 
not present in the results of the 18-month survey, though the overall values remain similar. This 
effect, though small, is strongly statistically significant, which suggests that it reflects a program 
impact that emerged in the time between surveys. Overall, however, the Family Rewards pro- 
gram had few temporary or lasting effects on parents’ health outcomes. 

Children’s Health Outcomes 

As discussed earlier in this report, the survey targeted “focal” children in three age cate- 
gories based on the grade at which they entered the study: elementary school (grade 4), middle 
school (grade 7), and high school (grade 9). In the 42-month survey, parents were asked to re- 
port on a set of questions related to the health outcomes of their focal child. These measures, 
some of which were introduced in the parent- focused tables, are presented in Appendix Table 
D.3, by child’s school level at the time of random assignment. 


45 Distress is measured using the Kessler Psychological Distress Scale, known as the “K-10 scale,” which 
is a 10-item questionnaire intended to yield a global measure of distress based on questions about anxiety and 
depressive symptoms that a person has experienced in the most recent four-week period; see Kessler et al. 
(2002). The ability to initiate and sustain action is assessed using the State of Hope scale, which measures goal- 
directed thinking; see Snyder et al. (1996). 

46 Riccio et al. (2010). 
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Elementary school students at the time of random assignment. For focal children in 
elementary school at the time of random assignment, there are no statistically significant differ- 
ences in overall health status or occurrence of health conditions. 

Middle school students at the time of random assignment. Health effects for 
children appear to be concentrated among those children who were in middle school at the time 
of random assignment. Program group respondents, on average, rated their focal child’s health 
higher than did those in the control group. They were more likely to rate their child’s health as 
5, or “excellent” (45.9 percent versus 38.3 percent, a statistically significant 7.6 percentage 
point increase), and less likely to rate it as 3, or “good” (21.2 percent versus 26.4 percent, a 
reduction of 5.3 percentage points). No differences in health conditions were observed between 
the control and program groups. 

High school students at the time of random assignment. There were no differences 
between high school-age children in the program and control groups in health status. Numbers 
remain largely consistent in magnitude across surveys for both the control and program groups. 
At the time of the 18-month survey, there were no program impacts on children’s health status or 
health conditions for elementary or middle school students. Program group high school students 
did, however, display statistically significant drops in any health condition and with asthma in 
particular. 47 At the time of the 42 -month survey, parents in the program group were, as noted 
above, more likely to report excellent health for their middle school-age children. This finding 
could represent an impact that emerged in the time between the surveys but, as it is not part of a 
larger pattern, caution is advised in interpreting this result. The effects on health conditions for 
high school students had dissipated by the time of the 42 -month survey. Overall, the Family 
Rewards program had few temporary or lasting effects on children’s health outcomes. 


Impacts for Key Subgroups 

Because average effects for the full group can mask program effects for some groups of study 
participants, the health impacts of Family Rewards are analyzed for subgroups defined by base- 
line measures of parental education, employment, poverty, and self-assessed health status. 48 


47 Riccio et al. (2010). 

48 MDRC conducted additional exploratory subgroup analysis on health impacts, by family type (single- or 
two-parent households), immigration status, reported risk factors for depression, food stamp and TANF receipt, 
and housing status (public housing. Section 8 vouchers, and other). Overall, few noteworthy effects were ob- 
served across these subgroups. The findings are included in Appendix H in the supplement to this report, Ric- 
cio et al. (2013), which is available at www.mdrc.org. Additional subgroup analyses were conducted on im- 
pacts on health insurance rewards earned for maintaining coverage, by TANF receipt. Again, few noteworthy 
effects were observed across the subgroups. These findings are included in Appendix Table D.4 at the back of 
this report. In addition to education and employment, variation in impacts was examined across a range of di- 

(continued) 
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These subgroups were pre-selected, based on the extensive literature demonstrating inequalities 
in health care access and health outcomes by socioeconomic characteristics. While numerous 
variables can capture the social and economic status of individuals, education, income, and em- 
ployment or occupational status are most often used to examine variations in the distribution of 
disease and health . 49 Recent reviews reveal that these measures remain persistent and pervasive 
predictors of variations in health . 50 

In general, impacts are expected to vary somewhat across subgroups, simply as a result 
of natural variation around the average impact for the full sample. This section focuses on 
whether that variation in impacts across subgroups is statistically significant, or whether it goes 
beyond what would be expected to occur naturally. For that reason, the focus is not on whether 
a given impact is statistically significant for the less educated subgroup, for example, but 
whether the difference between that impact and the impact for the more educated subgroup is 
statistically significant (as indicated by daggers in the rightmost column of the tables). If the 
difference between those two impacts is not statistically significant, the results suggest that the 
effects observed for the full sample generally hold across more and less educated individuals. 

Impacts on Parents’ Receipt of Health Care Services and Health 

Outcomes, by Educational Level at Baseline 

In general, studies find that more educated adults report better health than those who are 
less educated. They are less likely to be hypertensive, to suffer from diabetes, to report that they 
are in poor health, and to report anxiety or depression . 51 There are multiple reasons for those asso- 
ciations, although it is likely that such health differences are in part the result of differences in be- 
havior across education groups. Overall, research suggests very strong gradients where the better 
educated have more healthy behaviors along virtually every margin: those with more years of 
schooling are less likely to smoke, drink heavily, be overweight or obese, or use illegal drugs . 52 

The Family Rewards health impacts were analyzed separately for parents who had less 
than a high school diploma or GED certificate at the time of random assignment and for those 
with at least a high school diploma or GED certificate. The program’s impacts on a variety of 
health measures are shown in the supplement to this report . 53 For parents’ health outcomes, 
there are three statistically significant differences in impacts between the subgroups defined by 

mensions, including assisted housing status and poverty level at random assignment. Overall, the impacts are 
similar across these subgroups, with a few exceptions. 

49 Kreiger, Williams, and Moss (1997). 

50 See, for example, the “Social Environment Notebook” of the MacArthur Research Network on Socio- 
economic Status and Health, at www.macses.ucsf.edu/default.php. 

51 Muennig, Schweinhart, Montie, and Neidell (2009); Muennig (2007); Cutler and Lleras-Muney (2006). 

52 Muennig, Schweinhart, Montie, and Neidell (2009); Muennig (2007); Cutler and Lleras-Muney (2006). 

53 See supplementary Appendix Table H.8 in Riccio et al. (2013), available at www.mdrc.org. 
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possession of a high school diploma/GED certificate. 54 Program group members in the more 
educated subgroup show a statistically significant drop in high cholesterol versus the control 
group (7.8 percent versus 10.9 percent, a drop of 3.1 percentage points) while those in the less 
educated subgroup do not. Program group members in the less educated subgroup show a statis- 
tically significant increase in reported incidence of diabetes (15.2 percent versus 11.1 percent, 
an increase of 4.1 percentage points) while those in the more educated subgroup show no effect. 
The less educated subgroup also shows a higher impact on being treated for diabetes. This in- 
crease in reported diabetes may be the result of better screening by doctors during checkups, as 
program participants were provided with health checklists to discuss with their doctors. Overall, 
however, there are few differences in impacts by parents’ education level at the time of random 
assignment, suggesting that education was not an important factor in determining the health and 
health care effects of the Family Rewards intervention. 

Impacts on Parents’ Receipt of Health Care Services and Health 

Outcomes, by Employment Status at Baseline 

Several studies have documented the relationship between health and employment. 55 
Among studies of welfare recipients and low-income mothers, health problems (of parents and 
children) are a barrier to regular employment and to entry into the labor force. 56 These studies 
also show that employed women have consistently reported being in better health than have un- 
employed women. However, the causal chain underlying the relationship between employment 
and health is still open to debate. Longitudinal evidence seems to suggest that employment af- 
fects health, but reciprocal effects are also possible. 57 This section extends the subgroup analysis 
and examines the program’s impacts on health by respondents’ self-reported employment status 
at the time of random assignment. 

The supplement to this report includes impacts on parents’ receipt of health care ser- 
vices and on health outcomes, by employment status at random assignment. 58 Program partici- 
pants who were unemployed at the time of random assignment display a drop in obesity rate 
over the control group (46.6 percent versus 53.1, a drop of 6.5 percentage points), while those 
who were employed at random assignment display no statistically significant change. There are 
otherwise no significant differences in impacts by baseline employment status, suggesting that 
employment was not an important factor in determining the health and health care effects of the 
Family Rewards intervention. 

54 This includes only outcome measures that display statistically significant differences between the pro- 
gram and control groups. 

55 Rogot, Sorlie, and Johnson (1992); Ross and Mirowsky (1995). 

56 Zedlewski (1999). 

57 Polit, London, and Martinez (2001). 

58 See supplementary Appendix Table H.9 in Riccio et al. (2013), available at www.mdrc.org. 
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Impacts on Parents’ Receipt of Health Care Services and Health 

Outcomes, by Poverty Level at Baseline 

Though the same amounts of cash transfers were offered for successfully completing 
rewarded activities regardless of the participants’ poverty level, rewards for the poorest partici- 
pants represented a much larger gain in proportion to their base earnings. Put another way, 
while the absolute value of the rewards was fixed, the relative value of a reward payment varied 
according to participants’ poverty level. As the relative value of an incentive payment increases, 
the probability of trying to earn the reward may also increase. The very poor may have thus 
been more likely to work toward health rewards and therefore use health care services at a 
greater rate than participants who were less disadvantaged. This behavior could have led, in 
turn, to better health outcomes. Alternatively, for this population, greater poverty may make it 
more likely that a participant is covered by public health insurance. This likelihood results from 
the gap in Medicaid coverage: above a certain income a person may lose access to insurance 
from Medicaid but be unable to secure insurance through her employer. If that is the case, then 
less severe poverty may be associated with lower health insurance coverage. For those partici- 
pants who are living in less severe poverty, Family Rewards could have offered expanded ac- 
cess to health care services; those suffering more severe poverty would likely already have had 
access to preventive health care through Medicaid. Therefore, the former might instead evi- 
dence greater health impacts. 

To investigate these potential relationships, impacts on parents’ receipt of health care 
services and health outcomes were examined by parents’ income at random assignment. Par- 
ents with income at or above 50 percent of the federal poverty level and those with income 
below 50 percent of the federal poverty level were examined separately. Table 5.9 shows pro- 
gram impacts on parents’ health outcomes, by poverty level. There are almost no significant 
differences in health care and health outcomes by poverty level, suggesting that parental earn- 
ings, on average, were not a factor in determining the health care effects of the Family Re- 
wards intervention. 

Impacts on Parents’ Receipt of Health Care Services and Health 

Outcomes, by Self-Reported Health Status at Baseline 

The effectiveness of the Family Rewards program with regard to improving health out- 
comes and increasing the receipt of health care services may be related to the initial health status 
of its participants. If those who initially suffered from poor health were in that condition be- 
cause they lacked access to health care services relative to the larger sample, the Family Re- 
wards intervention could result in particularly improved health outcomes for this group. As 
part of the intervention, F amily Rewards also required that doctors fill out a checklist including 
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Table 5.9 


Impacts on Parents’ Receipt of Health Care Services and Health Outcomes, 
by Respondent’s Poverty Level at the Time of Random Assignment 



Program 

Control 

Difference 



Subgroup and Outcome 

Group 

Group 

(Impact) 

P-Value 

Sig. 

Income at or above 50% of FPL 
at baseline (%) 

Had a health checkup in past 12 months 

90.6 

90.1 

0.5 

0.770 


Had 1 or more dental checkups in past 
12 months 

83.1 

75.5 

J ^ *** 

0.002 


Had 2 or more dental checkups 

46.0 

35.0 

n o *** 

0.000 


Excellent or very good self-rated health 

41.3 

37.4 

3.9 

0.145 


Has any medical condition' 1 

51.3 

47.8 

3.5 

0.206 


Asthma 

15.1 

14.0 

1.0 

0.618 


High blood pressure/hypertension 

22.8 

23.5 

-0.7 

0.753 


High cholesterol/high LDL 

9.7 

9.6 

0.1 

0.968 


Diabetes 

9.3 

7.9 

1.4 

0.370 


Treated for any medical condition' 1 

46.3 

41.8 

4.5 * 

0.096 


Asthma 

13.1 

13.0 

0.1 

0.941 


High blood pressure/hypertension 

21.4 

21.5 

-0.1 

0.978 


High cholesterol/high LDL 

8.4 

7.1 

1.3 

0.379 

t 

Diabetes 

9.2 

7.2 

2.0 

0.208 


Obese according to Body Mass Index (BMl) b 

45.5 

44.1 

1.4 

0.630 


Sample size (total = 1,184) 

628 

556 




Income less than 50% of FPL 
at baseline 1%) 

Had a health checkup in past 12 months 

89.1 

87.2 

1.9 

0.411 


Had 1 or more dental checkups in past 
12 months 

78.6 

65.3 

13.3 *** 

0.000 


Had 2 or more dental checkups 

44.0 

31.3 

12.8 *** 

0.000 


Excellent or very good self-rated health 

27.3 

26.4 

0.9 

0.759 


Has any medical condition' 1 

61.3 

63.2 

-1.9 

0.549 


Asthma 

20.9 

19.6 

1.3 

0.649 


High blood pressure/hypertension 

34.9 

31.2 

3.7 

0.236 


High cholesterol/high LDL 

10.3 

14.4 

* 

o 

Tj - 

1 

0.082 


Diabetes 

14.7 

15.9 

-1.1 

0.649 



(continued) 
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Table 5.9 (continued) 


Subgroup and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Sig. 

Treated for any medical condition 1 

55.6 

57.2 

-1.6 

0.631 


Asthma 

18.0 

17.5 

0.5 

0.844 


High blood pressure/hypertension 

32.4 

29.6 

2.8 

0.364 


High cholesterol/high LDL 

9.4 

13.2 

-3.8 * 

0.087 

t 

Diabetes 

14.3 

15.3 

-1.1 

0.658 


Obese according to Body Mass Index (BMl) b 

46.8 

49.9 

-3.2 

0.392 


Sample size (total = 777) 

394 

383 





SOURCE: MDRC calculations using data from the Family Rewards 42-month survey. 


NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the differences between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 

5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical 
significance levels (Sig.) are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

FPL = federal poverty level. 

a The 4 most commonly reported conditions are listed. 

b Weight categories are from the National Institutes of Health. Obesity is defined as having a BMI 
of at least 30. About 6 percent of the sample is excluded from this analysis because of missing data. 


questions about common ailments during checkups (shown in Figure 5.1). Therefore, even if the 
rate at which individuals in the control and program groups remained comparable, the quality of 
care received may have improved for Family Rewards participants. This increase in quality 
could have been particularly important for those who were initially burdened with poor health. 
Furthermore, health status and employment are, as noted, correlated. Those in poor health may, 
of course, be less likely to obtain and sustain stable employment and thus may suffer financial- 
ly. Conversely, the material hardship and financial instability generated by unemployment can 
lead to increased physical and psychological stress, which can damage health. To the extent that 
program recipients suffered poor health as a result of stress induced by material hardship, Fami- 
ly Rewards may have improved health outcomes indirectly for this subgroup through its full 
range of conditional cash transfers. 

Table 5.10 shows impacts on parents’ receipt of health care services and health out- 
comes, by self-reported health status at the time of random assignment. The scale includes five 
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Table 5.10 


Impacts on Parents’ Receipt of Health Care Services and Health Outcomes, 
by Respondent’s Self-Rated Health at the Time of Random Assignment 



Program 

Control 

Difference 



Subgroup and Outcome 

Group 

Group 

(Impact) 

P-Value 

Sig. 

Self-rated health is excellent or verv good 






at baseline 1%) 






Had a health checkup in past 12 months 

89.4 

88.1 

1.3 

0.556 


Had 1 or more dental checkups in past 12 months 

82.9 

73.9 

g g H=H=H= 

0.002 


Had 2 or more dental checkups 

43.2 

35.0 

8.2 ** 

0.016 


Excellent or very good self-rated health 

53.2 

48.7 

4.5 

0.183 


Has any medical condition 1 

42.7 

39.8 

2.9 

0.381 


Asthma 

14.0 

10.4 

3.5 

0.117 

tft 

High blood pressure/hypertension 

17.6 

17.0 

0.7 

0.793 


High cholesterol/high LDL 

6.6 

7.4 

-0.9 

0.618 


Diabetes 

7.1 

5.5 

1.5 

0.347 


Treated for any medical condition 11 

37.0 

33.5 

3.5 

0.265 


Asthma 

11.8 

8.8 

3.1 

0.141 

fff 

High blood pressure/hypertension 

16.2 

15.6 

0.6 

0.794 


High cholesterol/high LDL 

5.0 

5.7 

-0.7 

0.658 


Diabetes 

7.0 

5.3 

1.8 

0.276 


Obese according to Body Mass Index (BMl) b 

43.0 

39.2 

3.8 

0.278 


Sample size (total = 847) 

425 

422 




Self-rated health is good at baseline 1%) 






Had a health checkup in past 12 months 

88.7 

89.5 

-0.8 

0.734 


Had 1 or more dental checkups in past 12 months 

82.9 

73.6 

g 3 H=H=H= 

0.003 


Had 2 or more dental checkups 

47.0 

35.2 

g H=H=H= 

0.002 


Excellent or very good self-rated health 

28.4 

29.1 

-0.7 

0.847 


Has any medical condition 1 

56.2 

54.4 

1.8 

0.617 


Asthma 

18.5 

14.5 

4.1 

0.149 

tft 

High blood pressure/hypertension 

28.2 

26.7 

1.5 

0.638 


High cholesterol/high LDL 

8.3 

11.4 

-3.1 

0.160 


Diabetes 

9.8 

10.9 

-1.1 

0.636 



(continued) 
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Table 5.10 (continued) 


Subgroup and Outcome 

Program Control Difference 
Group Group (Impact) 

P- Value 

Sig. 

Treated for any medical condition 3 
Asthma 

High blood pressure/hypertension 
High cholesterol/high LDL 
Diabetes 

Obese according to Body Mass Index (BMI) b 
Sample size (total = 702) 

52.5 

16.1 

26.7 

7.6 

9.6 

46.0 

372 

47.6 

13.6 
24.1 

8.7 

9.6 

49.9 

330 

4.9 

2.5 

2.7 

- 1.1 

- 0.1 

-3.9 

0.179 

0.354 

0.397 

0.595 

0.979 

0.321 

ttt 

Self-rated health is fair or Door at baseline (%) 






Had a health checkup in past 12 months 

93.5 

90.1 

3.4 

0.226 


Had 1 or more dental checkups in past 12 months 

75.1 

62.4 

12 7 *** 

0.008 


Had 2 or more dental check-ups 

45.7 

27.8 

1 7 g *** 

0.000 


Excellent or very good self-rated health 

12.6 

6.4 

6.2 ** 

0.039 


Has any medical condition 3 

80.0 

82.9 

- 2.8 

0.46 


Asthma 

21.8 

32.9 

.H I ** 

0.014 

ttt 

High blood pressure/hypertension 

45.9 

47.2 

-1.3 

0.801 


High cholesterol/high LDL 

20.6 

19.7 

0.9 

0.833 


Diabetes 

23.6 

23.4 

0.2 

0.961 


Treated for any medical condition 3 

73.1 

79.1 

- 6.0 

0.157 


Asthma 

19.4 

30.8 


0.009 

ttt 

High blood pressure/hypertension 

42.7 

46.1 

-3.4 

0.491 


High cholesterol/high LDL 

19.7 

18.6 

1.1 

0.790 


Diabetes 

22.7 

23.3 

-0.5 

0.901 


Obese according to Body Mass Index (BMI) b 

52.3 

54.2 

-1.9 

0.719 


Sample size (total = 398) 

219 

179 





SOURCE: MDRC calculations using data from the Family Rewards 42-month survey. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and control groups. 
The p-value indicates the likelihood that the differences between the program and control groups arose 
by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 

10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical significance 
levels (Sig.) are indicated as follows: ttt = 1 percent; ft = 5 percent; | = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 
a The 4 most commonly reported conditions are listed. 

b Weight categories are from the National Institutes of Health. Obesity is defined as having a BMI of 
at least 30. About 6 percent of the sample is excluded from this analysis because of missing data. 
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categories: excellent, very good, good, fair, and poor. For the purposes of this analysis, individu- 
als are grouped into three categories: fair or poor, good, and excellent or very good. Two impacts 
stand out. F irst, while those in the good and excellent or very good categories did not experience 
drops in common medical conditions relative to those in the control group, those in the fair or 
poor category did experience a drop in asthma (21.8 percent versus 32.9 percent, respectively, for 
the program and control groups — a difference of 1 1 . 1 percentage points). This difference is sta- 
tistically significant and significantly different from that of the other health subgroups. Second, 
program members in the fair or poor health category were 6.2 percentage points more likely to 
report excellent or very good health at the time of the survey (12.6 percent versus 6.4 percent). 
This impact is statistically significant; however, the impact for this subgroup category is not sta- 
tistically significantly different from the impact observed for the other two subgroup categories. 
This makes the observed variation in effects on this measure less certain. The drop in reported 
asthma symptoms among those with the worst self-reported health is an intriguing finding. It is 
possible that the apparent effect is illusory — in other words, that it is the product of statistical 
chance and not indicative of a “true” change. There is some reason, though, to think otherwise. 
First, the effect does not appear in isolation. Program group members in the fair or poor health 
category were also significantly more likely than control group members to report excellent or 
very good health at the time of the survey. Though the difference in impact between health sub- 
groups is not significant, an improvement in perceived health condition is consistent with de- 
creased asthma symptoms. Second, the asthma effect is significant at the 5 percent level. This is 
stronger than the 10 percent standard and makes it less likely that the effect is illusory. 

Subgroup analysis conducted separately for each of the three boroughs that the survey 
sample covered provides mixed results. The impact differential is present in the Bronx but not 
in Brooklyn or Manhattan. While this may indicate that the overall effect is simply the product 
of statistical noise, the much smaller sample sizes that were generated by dividing respondents 
according to borough reduces statistical power significantly. This makes it more difficult to de- 
tect present effects. 

If asthma symptoms diminished among program group members in this health sub- 
group, there should be evidence of a mechanism, an explanation for how participation in Family 
Rewards produced this result. Asthma symptoms are known to be correlated with smoking to- 
bacco and with physical and emotional stress. Family Rewards, as noted, required that partici- 
pants provide their doctors with a checklist of health-related questions to be answered and dis- 
cussed during a checkup. These included a question about use of tobacco products and 
instructions for the doctor to discuss options for quitting. Simply talking with one’s doctor about 
tobacco use and its risks might lead one to quit. 59 If Family Rewards led individuals in this 


59 Glynn (1990). 
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group to give up smoking, this could explain a drop in reported asthma symptoms. Furthennore, 
for this group, rewards for health, employment, and education activities may have reduced ma- 
terial hardship and financial stress, common triggers for asthma attacks. To explore potential 
mechanisms, MDRC conducted additional statistical analyses for a variety of outcomes for the 
health subgroups. MDRC examined the effects on smoking for the health subgroup but found 
no significant impact or impact differential. Turning to stressors, there were no significant im- 
pact differences by health subgroup for psychosocial well-being, a potential indicator of emo- 
tional stress. Finally, there were no significant impact differences by health subgroup for 
measures of material hardship and financial well-being, two indicators of financial stress. 

Overall, though both the impact on asthma rates that was detected for program group 
respondents in the fair or poor self-rated health subgroup and the difference in impact across the 
health subgroups are strong and statistically significant, there are no obvious changes in habits 
(for example, smoking) or stressors that might serve to explain them. However, a large decrease 
in the incidence of a debilitating condition for those who are already struggling with poor health 
remains an intriguing finding. Future research on conditional cash transfer programs with a 
health component should pay special attention to asthma. 


Conclusion 

Family Rewards health incentives were designed to encourage low-income families to adopt 
better preventive health care practices. It turned out that a higher proportion of families than the 
program’s designers had expected were already receiving health insurance coverage and prac- 
ticing preventive health care, which limited the program’s ability to improve some health prac- 
tices and health care behavior further for this sample. Nevertheless, Family Rewards produced 
small positive impacts on overall and continuous health insurance coverage and strong, lasting 
effects on the receipt of dental care by both parents and children. 

Given the already high rates of insurance coverage and health care receipt among study 
participants before random assignment, it is perhaps not surprising that the program did not 
produce many notable lasting effects on overall health outcomes for parents or children. 
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Chapter 6 

Work-Related Rewards and Effects 


The work and training incentives in Family Rewards are distinctive because, unlike incentives 
in many other employment and training- focused programs, they are part of a larger package of 
rewards that are intended to motivate positive family practices in education, health, and work. 
Earlier studies have found that providing work and training incentives in the absence of the 
other incentives can increase work and, in some cases, training. 1 Since the structure of Family 
Rewards allows families to earn rewards through a range of activities, effects on work and 
training depend not only on the labor market, but also on how families decide to complete 
health, education, and workforce-related activities to earn rewards. Parents and children can 
focus on some activities to the exclusion of others. Similarly, additional money that families 
receive from education and health rewards may reduce the need to work and offset some or all 
of the work incentive that these rewards offer. 

The first-year Family Rewards findings revealed a much lower receipt rate for work- 
force rewards than for the education and health rewards. As a result, and in preparation for the 
end of the three-year reward earnings period, the program made an effort to focus more on work 
during Year 2. MDRC’s 2010 report on Family Rewards, covering one year of unemployment 
insurance (UI) system data and 1 8 months of survey data from the time of random assignment, 
found that the program led to a small increase in employment in jobs that are not covered by the 
UI system and to a small decrease in employment in Ul-covered jobs. 2 

This chapter presents the effects of Family Rewards on parents’ employment and earn- 
ings over three years. The positive employment impact on survey-reported employment found 
on the 1 8-month survey is sustained after three years, while the small negative impact on UI- 
covered employment in Year 1 disappears in Year 2. It is not clear why program effects are 
different across Ul-covered and non-UI-covered employment. The survey data capture a 
broader range of jobs, since self-employment, federal government, domestic, and informal 
(including irregular or “off- the-books”) jobs are also not captured. 3 The earlier report suggested 
that non-UI-covered jobs may have been easier to obtain, more flexible, or more convenient. 

Evidence from reports of occupations and industries on the 42 -month survey gives cre- 
dence to this possibility. Survey-reported jobs that were not covered by UI records were 
overwhelmingly in service occupations, and most of those jobs were related to child care. Since 

'Riccio et al. (2008); Martinson and Hendra (2006); Michalopoulos et al. (2002). 

2 Riccio et al. (2010). 

3 Stevens (2007); Wallace and Haveman (2007). 
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most of the families in the study were female-headed households with multiple children, it 
makes sense that parents seeking to maintain full-time employment could more easily begin and 
sustain a job that might allow them to provide for their own child care needs, choose a conven- 
ient location, and work more flexible hours. 


The Work and Training Rewards Offer 

The schedule of workforce rewards is presented in Table 6.1. The first payment is for sustained 
full-time employment, requiring a participant to work at least 30 hours per week for six out of 
every eight weeks. Parents who work the minimum amount receive $300 every two months, or 
up to $1,800 per year. By increasing the payoff to work, the reward creates an incentive for 
parents to find full-time jobs, to move from part-time into full-time work, or to sustain full-time 
employment. For a parent working 40 hours per week at $8 per hour, for example, the reward 
effectively increases her net wage by 1 1 percent, to $8.90 per hour. 


The Opportunity NYC Demonstration: Family Rewards 

Table 6.1 


Work and Training Rewards 


Activity 

Payment 

Work 


30 hours or more per week for 6 weeks 
or more, each 2-month period 

$300 every 2 months 

Education and training 


Must work at least 10 hours per week, 
while completing an approved course 11 
Minimum work requirement discontinued 
in Year 3 

$300 to $600, per 8-week activity period, 
depending on length of course; maximum of 
$3,000 per adult over 3 years 


NOTE: a Training may also include basic education activities, such as English as a Second 
Language (ESL), basic skills, and General Educational Development (GED) preparation 
courses. 


Parents could earn another $300 to $600 every two months, up to a total of $3,000 dur- 
ing the program, if they completed an approved education or training course. In order to qualify 
for the education and training rewards during the first two years of the program, parents had to 
participate in approved education or training activities while working at least 10 hours per week. 
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Since the receipt of these rewards was so low in Years 1 and 2 (less than 2 percent of eligible 
adults earned these rewards), the minim um work hours requirement was discontinued in Year 3. 


Data and Samples 

The three data sources used in this chapter are Seedco’s Family Rewards program data, the 
Family Rewards 42 -month parent survey, and earnings records from the New York State 
unemployment insurance system. As described in Chapter 2, Seedco’s database tracked reward 
earnings and payments to every family in the program group. The rewards data cover the full 
three years that families were eligible for rewards (September 2007 through August 2010) and 
several months afterward to resolve late coupon payment submissions, correct bank account 
errors that were not allowing payments to be made, and reconcile reports of discrepancies 
between earned and paid amounts, through July 2011. 

The UI data provide quarterly employment and earnings information, as reported by 
employers, for the majority of workers in the state. These data are available for the entire 
Family Rewards evaluation sample of nearly 5,000 parents for several quarters before their 
date of random assignment and for three years afterward, and, unlike survey data, they do not 
depend on respondent recall. At the same time, although these records cover most employ- 
ment in the United States, they do not capture certain types of jobs, including self- 
employment, federal government employment, military jobs, informal jobs, and out-of-state 
jobs. Another drawback of UI records is that they do not provide information on hours 
worked during a quarter or week or on the characteristics of jobs held, such as hourly wage 
rates, benefits, and schedule. 

The Family Rewards 42-month survey, which was administered to participating parents 
about 42 months following random assignment, complements the UI data by capturing all types 
of jobs held and providing data on their characteristics. Respondents provided infonnation on 
whether these jobs were full time or part time, their wage rates, and any benefits offered by the 
employer. These data are available for a randomly chosen subset of the full evaluation sample, 
or just under 3,000 individuals. As discussed in detail in the supplement to this report, 4 although 
the survey sample is generally representative of the full evaluation sample, some caution must 
be used when interpreting findings from the survey. There were some differences in characteris- 
tics between program and control group respondents. (In particular, the program group had 
more earnings from Ul-covered jobs in the year before entering the study). 5 


4 See Appendix J in the supplement to this report, Riccio et al. (2013), which is available at www.mdrc.org. 

5 This difference is accounted for in the impact analysis. The impact regression model includes Ul-reported 
employment and earnings and a range of demographic characteristics. 
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Receipt of Work and Training Rewards 

While fewer families received education and health rewards in the third year of the program 
than they had during the first two years, 42 percent of families in the Family Rewards group, the 
same proportion as in previous years, earned at least one workforce reward in the program’s 
final year (as was shown in Table 2.3). Table 6.2 presents the receipt of the work and training 
rewards in more detail, focusing on parents rather than families as the unit of analysis. The top 
several rows show that, as in Years 1 and 2, almost all of the workforce rewards earned in Year 
3 were for full-time work. In addition, receipt of the workforce rewards was somewhat regular. 
Only about 1 in 1 0 parents who received a reward did so in only one activity period each year. 
In Year 3, nearly half of the parents who had earned any work-related reward did so in each 
activity period. Very few parents — only 4 percent of the program group — earned rewards for 
education and training at some point that year. 

Two recent employment studies — one evaluating a program in the United Kingdom 
and another in the United States — found that over 25 percent of participants who were offered 
financial incentives for education and training in those programs received them. 6 These em- 
ployment programs, however, had more extensive job coaching and referral services, which 
were not part of the Family Rewards program model. During interviews, program group 
members indicated that having no referral source and limited time to search for affordable 
training that could fit into their busy schedules on their own were challenges to earning training 
rewards. (See Box 6.1.) 

The receipt of education and training rewards remained low in Year 3, although the 
percentage of parents who earned these rewards rose from less than 2 percent in prior years to 4 
percent in Year 3. The concurrent 10-hour-a-week employment requirement for earning this 
reward was dropped in the final year of the program, creating an opportunity for families who 
were struggling to find work to earn more cash while acquiring more skills. Interviews with 
parents indicated that dropping this requirement in the last year of the program did not give 
families enough time to locate and complete training opportunities. (See Box 6. 1 .) 

Receipt rates by employment status at baseline may provide a better indication of indi- 
viduals’ ability or willingness to earn rewards for full-time work, as shown in the bottom panel 
of Table 6.2. Not surprisingly, receipt rates are much higher for the 41 percent of parents who 
reported being employed full time (or for at least 30 hours per week) when they entered the 
study, a group most likely to be eligible for the rewards and consequently most likely to be 
earning those rewards as a windfall. About 73 percent of parents in this group received at least 


6 Hendra et al. (2011); Miller, van Dok, Tessler, and Pennington (2012). 
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The Opportunity NYC Demonstration: Family Rewards 

Table 6.2 

Workforce Rewards Earned by Parents 






Years 1, 2, and 3 

Outcome 

Year 1 

Year 2 

Year 3 

Combined 

Full program group (%) 





Parent earned at least one workforce reward 

41.6 

41.5 

41.2 

52.6 

For full-time employment d 

41.1 

41.3 

39.3 

51.2 

For education and training while employed b 

1.8 

1.9 

4.0 

5.5 

Parent earned a workforce reward in L (%) 





1 activity period only 

12.9 

10.6 

11.2 

- 

2 or 3 activity periods only 

21.6 

17.4 

17.7 

- 

4 or 5 activity periods only 

27.8 

28.2 

24.3 

- 

6 activity periods 

37.7 

43.8 

46.8 

— 

Sample size 




2,515 

Bv employment status at random assignment (%) 

Parent earned at least one full-time employment reward d 





Employed full time at random assignment e 

73.2 

69.7 

65.8 

82.3 

Employed part time at random assignment 

40.8 

42.7 

43.2 

57.3 

Not employed at random assignment 

12.9 

15.8 

14.9 

22.5 

Sample size 




2,322 


SOURCE: MDRC calculations using Seedco's Family Rewards program data. 

NOTES: The first program year covers September 2007 through August 2008; the second program 
year covers September 2008 through August 2009; and the third program year covers September 
2009 through August 2010. 

A double dash (— ) indicates "not applicable." 

At the time of random assignment, 967 program group members reported being employed full 
time, 206 reported being employed part time, and 1, 149 reported not being employed. 

a Full-time employment is at least 30 hours per week for 6 weeks or more in each 2-month 
payment period - that is, approximately 75 percent of the time. 

b ln Years 1 and 2, a parent had to work at least 10 hours per week, while attending a course of at 
least 35 hours. In Year 3, the 10-hour work requirement was discontinued. 
c This measure is calculated only for adults with any workforce rewards. 
d A total of 8 percent of the sample was missing employment status information at the time of 
random assignment. 

e Full-time employment at random assignment refers to working at least 30 hours per week. 
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Box 6.1 

Hurdles to Earning Training Rewards 


In the qualitative interviews that were conducted after the first year of the program, many 
parents expressed regret about not being able to claim the work-related training reward. The 
reasons for their limited ability to earn those rewards ranged from physical and logistical 
challenges to misconceptions about the reward’s requirements. Examples include: 

• Not being able to successfully locate an affordable, approved course or part-time job 
that fit into their schedules, especially given child care responsibilities and the fact that 
most were searching for resources on their own 

• Not being able to demonstrate completion of an approved course because of difficulties 
obtaining the necessary documentation 

• Believing that the training they were receiving through the welfare office did not quali- 
fy for the reward 

• Eironeously thinking that the part-time work requirement had not been eliminated in 
the third year or believing that they had missed the deadline for submitting training 
documentation 

• Health issues that prevented them from either attending training or working part time 

One parent expressed strong frustration that the part-time work requirement was eliminated 
so late in the program. She felt that by the third year, participants did not have adequate lead 
time to find and enroll in training: “If they had just had training [from the beginning], a lot of 
families would have got out more and maybe be working by now [Year 3].” She went on to 
explain the challenges of balancing framing and work for mothers in her community: 

It was hard for a . . . lot of families to jump from training and part-time work be- 
cause you know what it did? A lot of training was from 9:00 to 5:00, and so they 
had to work nighttime, so it was hard. Then you have to get a babysitter, so it 
didn’t work for a lot of families. 


one reward during the first program year. The receipt rate for full-time workers at study entry 
fell each year, to 66 percent by Year 3. The large drop in the proportion of those parents who 
earned rewards for full-time work may be in part a result of the economic recession that hit 
toward the beginning of Year 2. In contrast, parents who were not working or were working 
part time at study entry had slightly higher receipt rates in Years 2 and 3 than in the first year of 
the program. Their rates, however, were still lower overall than the rates of those who entered 
the program as full-time workers. 
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Impacts on Education and Training 

This section presents the program’s effects on participation in education and training activities 
and on certificate and degree receipt. The previous report on Family Rewards found no overall 
effect of the program on education or training participation rates through 1 8 months, although 
the program did lead to small increases in credential receipt. 7 

Table 6.3 presents effects on participation in education, training, and employment- 
related activities. By the 42 -month point, 41percent of respondents in the control group reported 
having participated in some activity since study enrollment. This rate is not much higher than 
the rate at 1 8 months, suggesting that not many additional individuals took up training after that 
point. Within 42 months, the program led to a small increase in participation in college courses: 
15 percent of the program group had taken at least one college course for credit, compared with 
13 percent of the control group. At the time of the survey, program group parents were also 
more likely to be in some kind of education or training activity than were control group parents. 

Table 6.4 presents data on credential receipt. The table shows the percentage of individ- 
uals holding a given credential as of the 42 -month survey interview point, so these numbers 
include both adults who obtained these credentials after they enrolled in Family Rewards and 
adults who held these credentials before they entered the study. However, any difference 
observed between the groups would have occurred since random assignment and can be 
attributed to the program. Similar to the findings from the previous report, the program led to 
small but statistically significant increases in the proportion of parents with a trade license or 
certificate (by 4 percentage points) and in the proportion of parents who held an associate’s 
degree (by 2 percentage points). Data on the type of license or certificate that was received 
indicate that a large fraction of these credentials were for service occupations, including home 
health aides, nursing, and child care. 

The training rewards are unlikely to have led to such effects on participation and cre- 
dential receipt, because so few individuals received those rewards. Although it is not certain, 
effects on participation and credential receipt may instead be driven by the extra income 
provided to families through the program, which may have functioned as a type of financial aid 
to either cover tuition or offset necessary child care or transportation costs associated with 
attending classes. 


Impacts on Employment and Earnings 

This section presents the effects of Family Rewards on employment and earnings over the three 
years that families were in the study. Effects for the subsample of parents who responded to the 


Riccio et al. (2010). 
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The Opportunity NYC Demonstration: Family Rewards 

Table 6.3 


Impacts on Participation in Employment and Education Activities 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Ever participated in any education, training. 

or employment activity (%) 

42.2 

41.2 

1.0 

0.577 

ABE, GED, or high school classes 

9.8 

10.1 

-0.3 

0.748 

ESL classes 

7.9 

7.9 

0.0 

0.965 

College courses for credit 

14.9 

12.6 

2.3 * 

0.060 

Vocational training 

15.9 

16.1 

-0.3 

0.838 

Other educational, training, or employment 

program activities 

8.5 

8.5 

0.1 

0.956 

Currently participating in any education, training. 

or employment-related activity (%) 

13.2 

10.1 

3 1 *** 

0.008 

Sample size (total = 2,966) 

1,543 

1,423 




SOURCE: MDRC calculations using data from the Family Rewards 42-month survey. 

NOTES: A two-tailed t-test was applied to differences between outcomes for the program and 
control groups. The p-value indicates the likelihood that the difference between the program and 
control groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 
percent; ** = 5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. 

Rounding may cause discrepancies in calculating sums and differences. 

Percentages may sum to more than the overall participation rate because sample members could 
list more than one response. 

ABE = Adult Basic Education. GED = General Educational Development. ESL = English as a 
Second Language. 


42-month survey are examined using survey data. As mentioned in earlier chapters, all survey 
respondents were interviewed after the three-year period during which they were eligible to earn 
rewards, but because some payments lagged, a small percentage of the subsample responded to 
the survey before receiving their last reward payments. As a result, the work-related impacts for 
the survey sample reflect the early post-program period — a time when families were adjusting 
to no longer receiving income from the program. 

Effects for the full sample of parents are estimated using UI records data. Because the 
data are reported quarterly, the follow-up for most sample members (who entered the study 
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The Opportunity NYC Demonstration: Family Rewards 

Table 6.4 


Impacts on Educational Attainment 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Has any degree, license, or certificate (%) 

82.6 

82.2 

0.4 

0.723 

Has any trade license or training certification (%) 

54.4 

50.3 

4.2 ** 

0.021 

Home health aide 

11.3 

10.5 

0.8 

0.451 

Nurse's aide/nurse's assistant (CNA) 

4.7 

4.4 

0.3 

0.665 

Child care/teaching 

6.3 

5.0 

1.4 

0.114 

Security 

3.5 

3.7 

-0.2 

0.750 

Other 

28.6 

26.6 

2.0 

0.232 

Has any degree or diploma (%) 

66.7 

66.5 

0.2 

0.849 

Highest degree or diploma (%) 

GED certificate 

18.7 

20.9 

-2.1 

0.140 

High school diploma 

27.3 

28.7 

-1.4 

0.365 

Associate's degree 

10.8 

8.4 

2.4 ** 

0.022 

Bachelor's degree or higher 

9.8 

8.4 

1.4 

0.169 

Sample size (total = 2,966) 

1,543 

1,423 




SOURCE: MDRC calculations using data from the Family Rewards 42-month survey. 


NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the difference between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; **= 5 
percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

This table reports on degrees, licenses, and diplomas received, regardless of whether they were 
received before or after random assignment. 

GED = General Educational Development. CNA = Certified Nurse's Assistant. 


during July through September of 2007) covers July 2007 through September 2010. As dis- 
cussed below, the survey data typically capture more employment than UI records capture, 
since the latter data source only includes employment reported to the UI system. 

It is important to consider these findings in the context of two major factors that might 
have affected work activity among the study sample. First, the tough economy in the wake of 
the recession that began during Year 2 of Family Rewards may have made finding and keeping 
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a job more challenging. The recession, although external to the program, could affect the 
program and control groups in different ways. For example, individuals in Family Rewards may 
have been less likely to search for a job if the cash rewards in the health and education domains 
provided more financial security when jobs were scarce. The second factor, which might make 
individuals in the program more likely to seek work, was the prospect that the cash payments 
would stop after the third program year. In preparing for the end of the program, families may 
have increased their employment activity in an effort to maintain their cash flow. 

Results Based on the Survey 

The previous report on Family Rewards showed that the program had increased survey- 
reported employment by 3 percentage points relative to the control group’s rate over the first 18 
months of follow-up and by 5 .6 percentage points at the time of the 1 8-month survey. Nearly all 
of the new employment was in full-time work. More than half of these jobs provided benefits 
such as paid leave and health insurance, and about a third of these jobs were reported as self- 
employment. 8 

Table 6.5 presents data on employment and current job characteristics within the 42 
months following random assignment. 9 At the 42 -month survey point, 50 percent of parents in 
the control group reported that they were working and earning, on average, about $14 an hour. 
The majority of the employed sample was working full time (30 hours or more per week), and 
over half of these workers received key benefits. The control group’s employment and job 
characteristics at 42 months were very similar to that reported at 1 8 months, with the exception 
that the overall employment rate was somewhat higher at 18 months, when 54 percent of 
respondents reported being employed. 10 

At the 42 -month point, Family Rewards increased self-reported employment by 6 per- 
centage points, from 50 percent for the control group to 56 percent for the program group. The 
program similarly increased employment in the prior year, by 4 percentage points. Employment 
impacts grew with the longer follow-up period, mainly because of a larger drop in the current 
employment rate among the control group than among the program group. 

Although Family Rewards produced strong employment impacts, about 44 percent 
of the program group was not employed at the time of the survey interview, and about 10 


s Riccio et al. (2010). 

Respondents who were not employed at the time of the survey are included in all the current job charac- 
teristic measures except for the average hourly wage. The average hourly wage measure includes only respond- 
ents who were employed at the time of the survey. 

10 Riccio et al. (2010). 
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Table 6.5 


Impacts on Employment and Job Characteristics 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Employment status (%) 





Currently employed at the time of the survey 

56.0 

49.6 

6 4 *** 

0.000 

Employed in past year 

66.2 

61.8 

4 3 *** 

0.004 

Characteristics of current iob a 





Average hourly wage ($) 

13.79 

14.02 

— 

— 

Less than $7.00 (%) 

6.5 

4.4 

2.1 ** 

0.011 

$7.00 - $8.99 (%) 

8.5 

7.6 

0.9 

0.365 

$9.00 or more (%) 

35.0 

30.2 

4 $ *** 

0.001 

Not reported (%) 

6.1 

7.5 

-1.4 

0.135 

Hours worked per week (%) 





1-19 

3.4 

3.2 

0.3 

0.693 

20-29 

7.0 

5.9 

1.2 

0.201 

30-34 

6.1 

6.0 

0.0 

0.979 

35 or more 

38.3 

33.5 

4 9 *** 

0.001 

Not reported 

1.2 

1.1 

0.1 

0.805 

Worked at least 30 hours per week (%) 

44.4 

39.5 

4 9 *** 

0.001 

Average weekly earnings ($) 

257 

226 

31 3 *** 

0.001 

Usual work schedule (%) 





Regular daytime shift 

40.7 

37.4 

3.3 ** 

0.037 

Regular evening/night shift 

6.8 

5.4 

1.4 

0.112 

Rotating or split shift 

4.9 

3.9 

1.0 

0.196 

Irregular shift 

2.9 

2.1 

0.8 

0.178 

Other 

0.6 

0.7 

- 0.1 

0.794 

Self-employed (%) 

7.1 

4.9 

2.2 ** 

0.011 

Employer-provided benefits ’ 5 (%) 





Paid sick days 

33.6 

30.4 

3 2 ** 

0.026 

Paid vacation days 

35.3 

33.3 

2.0 

0.152 

Paid holidays, including Christmas 





and New Year's Day 

35.8 

32.7 

3.1 ** 

0.029 

Dental benefits 

27.1 

24.7 

2.4 * 

0.083 

A retirement plan 

28.1 

25.1 

29 ** 

0.033 

A health or medical insurance plan 

29.5 

26.9 

2.5 * 

0.066 

Enrolled in a work-related health or 





medical insurance plan 

21.3 

20.2 

1.1 

0.393 

(continued) 
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Table 6.5 (continued) 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Employment search (%) 





Looked for work in previous 4 weeks 

24.6 

25.3 

-0.7 

0.638 

Sample size (total = 2,966) 

1,543 

1,423 




SOURCE: MDRC calculations using data from the Family Rewards 42-month survey. 


NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the differences between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 
5 percent; * = 10 percent. 

Italic type indicates comparisons that are nonexperimental. Statistical tests were not performed. 
Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 
a lf a respondent worked multiple jobs at the time of the interview, then only the characteristics 
of the primary job are reported. (The job at which the respondent worked the most hours is 
considered primary.) Respondents who were not employed at the time of the survey are included in 
all the current job characteristics measures, except for average hourly wage. The average hourly 
wage measure includes only respondents who were employed at the time of the survey. 

b This includes benefits that are or eventually will be offered, regardless of whether the 
respondent received them. 


percent of the program group was working part time. (The data discussed in this paragraph and 
the next are not shown in Table 6.5.) These parents would not have been eligible to earn the 
full-time work rewards and may have decided to pursue more education as a result. Although 
that may be true for some, survey responses indicate that, for the most part, that was not the 
case. Parents who were not employed at the time of the survey indicated that the main reasons 
they did not work were their own or a family member’s illness or disability (46 percent) and 
their inability to find work (22 percent). Only 4 percent of those who were not working cited 
school and training as their main reason for not working. 

Parents who had part-time work at the time of the survey were mainly unable to find 
full-time work or increase their current working hours. About 49 percent of program group 
members who were working part-time indicated having difficulty finding full-time work. For 
this group, education and training was a more prominent reason for not working full-time than 
for the unemployed group (13 percent indicated school and training as their main reason, 
compared with 4 percent of those who were not working), as was taking care of their homes and 
families (15 percent, compared with 12 percent of those who were not working). 
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Family Rewards also led to impacts on a range of job characteristics. The program in- 
creased full-time employment, employment in jobs with regular daytime shifts, and employ- 
ment in jobs with a range of key benefits. These effects indicate that much of the increase in 
employment that the program generated was in full-time jobs that offered benefits. The program 
also increased earnings by about $32 a week, 11 with the impact primarily driven by the higher 
employment rate in the program group. 

Consistent with findings at the 1 8-month point, which was administered about halfway 
through the program, about one-third of the increase in survey-reported employment was in 
self-employment (derived by dividing the 2.2 percentage point increase in self-employment by 
the 6.4 percentage point increase in total employment). 12 The proportion of new employment 
attributed to self-employment is notable, since only 7 percent of the program group and 5 percent 
of the control group reported being self-employed. 

Results Based on Administrative Records 

Table 6.6 presents impacts on Ul-reported employment and earnings for the three- 
year follow-up period for the full sample. Considering outcomes for the control group first, 
about 65 percent of parents in the control group worked at some point during the follow-up 
period in Ul-covered jobs. On average, they were employed in about half of the follow-up 
quarters and earned just over $12,000 per year. This average earnings amount includes zeroes 
for those who did not work; it does not represent the actual earnings of just those who 
worked. Among those who did work, average earnings were about $19,000 a year (not shown 
in table). Even though the nationwide recession occurred in Year 2 and lasted through Year 3 
of the follow-up period, it did not seem to cause employment rates of the Family Rewards 
sample to fall dramatically in later years. 

The earlier report showed that the program reduced Ul-covered employment slightly in 
the first year of Family Rewards. 13 The top panel of Table 6.6 presents the impacts for three 
follow-up years. The program group was slightly less likely to have worked than the control 
group in Year 1, with a negative impact of 2.4 percentage points. By Year 2, this negative effect 
was no longer statistically significant. The earlier report considered the possibility that the 
program created a disincentive to work in Ul-covered employment, as a result of what economists 
call the “income effect” — a reduction in work in response to increased income from sources 


"in the table, the hourly wage is measured among those who worked, while weekly earnings are meas- 
ured among the survey sample. 

12 The proportion of new jobs that have certain characteristics can be determined by dividing the program’s 
impact on a particular characteristic by its impact on the employment rate. 

°Riccio et al. (2010). 
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Table 6.6 

Impacts on Ul-Covered Employment and Earnings, Years 1 to 3 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Ever employed (%) 

Years 1-3 

63.3 

64.9 

-1.6 

0.114 

Year 1 

56.3 

58.7 

_2 4 *H=H= 

0.008 

Year 2 

55.0 

56.2 

-1.2 

0.240 

Year 3 

52.5 

53.3 

-0.9 

0.420 

Average quarterly employment (%) 

Years 1-3 

47.7 

48.7 

-1.0 

0.209 

Year 1 

49.1 

50.4 

-1.3 * 

0.098 

Year 2 

47.8 

48.8 

-1.0 

0.273 

Year 3 

46.1 

46.7 

-0.6 

0.512 

Total earnings ($) 

Years 1-3 

36,912 

37,506 

-594 

0.424 

Year 1 

12,154 

12,376 

-221 

0.323 

Year 2 

12,344 

12,601 

-257 

0.369 

Year 3 

12,414 

12,529 

-116 

0.720 

Sample size (total = 4,993) 

2,513 

2,480 




SOURCE: MDRC calculations using data from New York State unemployment insurance (UI) 
wage records. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the difference between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; 
** = 5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. Standard errors were adjusted to account for 
multiple observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Dollar averages include zero values for sample members who were not employed. 

This table includes only employment and earnings in jobs covered by the New York State 
Ul program. It does not include employment outside of New York State, nor in jobs not covered 
by the UI system (for example, "off-the-books" jobs and federal government jobs). 


other than work (from the education and health rewards in this case). However, these longer- 
term results suggest that any slight reduction of work or earnings that was detected for the full 
sample early in the program was not sustained over time. It is possible that the workforce 
incentives caused families who would otherwise have reduced their work effort to maintain or 
acquire more Ul-covered work as their economic outlook became more uncertain, because of 
the recession, or as they approached the end of the program. 
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Comparison of Results from the Survey and Administrative Records 

Even though survey data often capture jobs that are not covered by UI data, it is not 
immediately clear why survey findings show a large, positive employment effect that does not 
appear in the UI data. For low-income populations, infomial work and self-employment are 
typically more prevalent and probably do account for a significant fraction of the jobs that are 
not reflected in the UI records. 14 The new, program-generated employment captured by the 
survey may include self-employment and a few “off-the-books” jobs but may also reflect 
federal government and out-of-state jobs, 15 which UI records do not cover. 

Table 6.7 compares the employment rate reported on the survey with employment that 
appears on UI records for survey respondents in the quarter of their interviews. The program led 
to a 6.4 percentage point increase in survey-reported employment but did not affect Ul-covered 
employment. The employment rates for the control group are consistent, at just under 50 
percent according to both sources. In contrast, the survey shows an employment rate that is 9 
percentage points higher than what UI records indicate for the program group. Some, but not all, 
of the new employment captured on the survey can be attributed to self-employment. Very few 
parents — only 1 percent of both the program and control groups — had out-of-state jobs that 
would not show up in UI records. 

Since the Family Rewards program offered incentives for all types of full-time jobs, it is 
not clear why the program would only increase non-UI-reported employment. It is unlikely that 
all of the increase in survey-reported employment is in “off-the-books” jobs, since individuals 
were required to show evidence of income tax withholding or other employer-related tax 
documentation in order to earn the full-time work rewards. 16 The impacts described earlier on 
earnings, work schedules, and benefits also do not indicate movement into many off-the-books 


14 Other research suggests that the UI data may miss relatively more employment for low-income populations 
than for higher-income groups. For example, the survey-Ul discrepancy is larger for less-educated workers; see 
Abraham, Haltiwanger, Sandusky, and Spletzer (2009). In addition, smaller employers and employers with high 
turnover, who tend to employ relatively high numbers of less skilled workers, tend to underreport earnings to the 
UI system more than do other types of employers; see Burgess, Blakemore, and Low (1998). 

15 Because New Jersey and Connecticut are easily accessible to residents of New York City and its sur- 
rounding boroughs, it is fairly common for residents of New York to be working outside the state. 

16 Family Rewards did not reward frill-time work in “off-the-books” jobs. To make sure individuals did not 
receive reward payments for unapproved jobs, the program’s documentation requirements for full-time work were 
extensive. Program group members who held a regular frill-time job had to submit copies of their pay stubs to 
earn the full-time work reward. If they did not receive pay stubs, they needed to fill out an Employment Verifica- 
tion Form and submit copies of their paychecks. Self-employed individuals needed to submit Quarterly Estimated 
Tax Payments (Form 1040ES) or Annual Tax Payments for any of the months for which they submitted coupons 
to receive rewards, as well as their tax returns with the Profit and Loss from Business Statement (Schedule C). 
Additional forms and timesheets were required for home-based or out-of-home child care jobs. 
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Table 6.7 


Employment and Earnings Impacts Based on Unemployment Insurance 
Administrative Records and 42-Month Survey Data 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

42-month survey respondents 





Currently working, according to survey (%) 

56.0 

49.6 

6 4 *** 

0.000 

Working in quarter of survey interview, 
according to UI data (%) 

46.7 

45.2 

1.4 

0.324 

Self-employed, according to survey (%) 

7.1 

4.9 

2.2 ** 

0.011 

Working out of state, according to survey (%) 

1.3 

1.4 

-0.1 

0.915 

Sample size (total = 2,966) 

1,543 

1,423 




SOURCES: MDRC calculations using data from the Family Rewards 42-month survey and New York 
State unemployment insurance (UI) administrative records. 


NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the difference between the program and control groups 
arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; 

* = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

jobs. The documentation process may also have prompted some parents to legitimize their 
previously infonnal employment. 

Survey response bias might be a concern for these employment outcomes. Less disad- 
vantaged sample members are often more likely to respond to a survey about the program than 
sample members who are more disadvantaged. In the case of Family Rewards, program group 
respondents were much more likely to earn a work-related reward in the final year of the 
program than were program group nonrespondents . 17 However, because, according to UI wage 
data, impacts for the respondent sample do not look different from the impacts for the full 
sample, response bias is not likely to have distorted impact differences between UI and survey- 
reported employment. 


1 Additional details are provided in the response bias analysis in Appendix J, which appears in the sup- 
plement to this report, Riccio et al. (2013), at www.mdrc.org. 
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The earlier report looked in a bit more detail at UI versus non-UI employment, and 
showed that the non-UI jobs were “lower quality” than UI jobs in that they paid somewhat less 
and were less likely to offer key benefits. 18 Additional details on the types of non-UI-covered 
jobs that parents held at the time of the 42 -month survey appear in Appendix Table E.6. 

That table also compares the occupations and industries of individuals who reported 
working at the time of their survey interviews but were not shown as working in the UI data 
with individuals who were employed according to both the survey and UI data. 

Although jobs in the services sector were common among all employed adults in the 
study, such positions were more prevalent among people working in non-UI-covered jobs 
(61 percent) than among people working in Ul-covered jobs (47 percent). Moreover, 35 
percent of people with non-UI jobs worked in child care, whereas only 2 percent of those 
with Ul-covered jobs were in that field. In contrast, people with Ul-covered jobs were more 
likely to work in the health care field, in positions such as home health aides and nurse’s 
aides or assistants. 

In summary, health care jobs made up a large part of the employment that was both 
reported on the survey and captured by the UI data. In contrast, child care jobs made up 
more than a third of the jobs that were reported on the survey but missed by the UI data. 
This pattern suggests that part of the reason why the survey data indicate that Family 
Rewards increased employment while the UI data do not show that same effect may be 
because child care jobs, which the UI system is less likely to cover, account for a portion of 
the increase in employment on the survey. It is not surprising that parents looking for full- 
time work might seek jobs in child care. Because most of the adults in the study are single 
mothers with more than one child, child care may have been an easy way to serve the dual 
purpose of caring for their own children and earning money by providing care for other 
working parents. 


Impacts for Key Subgroups 

Family Rewards produced significant gains for the full sample in employment in non-UI- 
covered jobs but did not affect employment in Ul-covered jobs during the program years. Often, 
however, average impacts for the full sample can mask variation in effects across subgroups. 
The earlier Family Rewards report found different effects on employment across a range of 
subgroups, including subgroups based on self-reported employment status and educational 
attainment at study entry. This section updates those effects by employment status and educa- 
tional attainment, and presents effects by poverty level. 

ls Riccio et al. (2010). 
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It is easy to imagine that the effects of the program might vary across subgroups defined 
by dimensions of employment, education, and poverty level. For example, adults who are 
already employed when they enroll in a program that offers an incentive for working full time, 
compared with adults not working at that time, may be less likely to increase their earnings in 
response to the rewards because many of them may be working full time already. In fact, 
Appendix Table E.7 shows that, among the control group members, parents who were employed 
at the time of random assignment earned about six times more in Ul-covered jobs over three 
years than did parents who were not employed at the time of random assignment ($62,000 
compared with $10,000). Table 6.8 and Appendix Table E.8 show the same pattern of large 
differences in three-year UI earnings between more and less “work-ready” subgroups. Although 
there is less room for improvement among individuals who are already employed, many low- 
income workers do not work steadily, or work only part time. 19 In contrast, there is more room 
for improvement among individuals who were not working at random assignment, but they may 
also face greater barriers to moving into full-time work in order to earn the rewards. The earlier 
report found some differences in impacts across these groups, with a general pattern of positive 
effects on survey-reported employment for the more work-ready subgroups and negative effects 
on Ul-covered work for the less work-ready subgroups. 20 Table 6.8 and Appendix Table E.7 
update these findings. 

This pattern of differential employment and earnings effects between more and less 
“work-ready” subgroups defined by prior employment, education credentials, and income levels 
persisted after three years of Family Rewards. Appendix Table E.7 presents impacts by em- 
ployment status at baseline. According to the 42 -month survey, Family Rewards produced an 
increase of 7 percentage points in self-reported employment and 4 percentage points in Ul- 
covered employment for those in the survey sample who were working at baseline and, accord 
ing to UI records, it had no effect on average quarterly Ul-covered employment. In contrast, the 
program had a smaller effect on employment according to the survey and a pattern of small 
negative (but not all statistically significant) effects on UI employment for those who were not 
working at baseline. There were no significant differences in impacts on employment or 
earnings between full-time and part-time workers. (See the supplement to this report.) 21 


19 Martinson and Hendra (2006) also recently found that a post-employment program that included a finan- 
cial incentive for full-time work had larger effects for individuals who had been employed in the year before 
random assignment, indicating that those who were more recently in the labor market could more effectively 
respond to an incentive to reenter it. 

20 Riccio et al. (2010). 

21 See Appendix Table 1.7 in the supplement to this report, Riccio et al. (2013), which is available at 
www.mdrc.org. 
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Table 6.8 


Impacts on Employment, Earnings, and Average Rewards Earned, 
by Respondent’s Education Level at the Time of Random Assignment 


Subgroup and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Sig. 

High school diploma/GED certificate 
or higher at baseline 






Full sample, UI records 

Average quarterly employment, Years 1-3 (%) 

56.3 

55.8 

0.4 

0.682 

tt 

Total earnings. Years 1-3 ($) 

48,320 

48,406 

-86 

0.937 


Sample size (total = 2,863) 

1,404 

1,459 




Survey sample 






Self-reported responses 






Currently working (%) 

63.8 

55.4 

8.3 

*** 0.000 


Ever worked since random assignment (%) 

82.9 

81.2 

1.7 

0.278 


Number of months worked in past year 

7.6 

6.8 

0.8 

*** 0.000 

t 

Ul records 






Working in quarter of survey interview, 
according to UI (%) 

55.6 

53.2 

2.5 

0.202 


Average quarterly employment. Years 1-3 (%) 

57.1 

55.8 

1.2 

0.345 

tt 

Total earnings, Years 1-3 ($) 

48,164 

46,740 

1,424 

0.287 

t 

Sample size (total = 1,769) 

915 

854 




Rewards earned, program group families 

Amount of education rewards earned ($) 

4,095 





Amount of health rewards earned ($) 

3,146 

- 

- 



Amount of workforce rewards earned ($) 

2,321 

- 

- 



Total rewards earned ($) 

9,563 

- 

- 



Sample size 

1,341 






(continued) 


179 






Table 6.8 (continued) 


Subgroup and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Sig. 

No high school diploma/GED certificate 
at baseline 






Full sample, UI records 

Average quarterly employment. Years 1-3 (%) 

36.2 

39.2 

-3.1 ** 

0.014 

tt 

Total earnings, Years 1-3 ($) 

20,730 

22,519 

-1,790 * 

0.060 


Sample size (total = 1,960) 

1,021 

939 




Survey sample 






Self-reported responses 






Currently working (%) 

44.0 

40.8 

3.2 

0.203 


Ever worked since random assignment (%) 

69.4 

68.0 

1.4 

0.568 


Number of months worked in past year 

5.3 

5.2 

0.1 

0.601 

t 

UI records 






Working in quarter of survey interview, 
according to UI (%) 

33.1 

33.0 

0.1 

0.973 


Average quarterly employment, Years 1-3 (%) 

34.6 

37.9 

-3.3 ** 

0.042 

tt 

Total earnings, Years 1-3 ($) 

18,727 

20,805 

-2,078 * 

0.081 

t 

Sample size (total = 1,128) 

592 

536 




Rewards earned, program group families 

Amount of education rewards earned ($) 

3,570 





Amount of health rewards earned ($) 

2,735 

- 

- 



Amount of workforce rewards earned ($) 

1,083 

- 

- 



Total rewards earned ($) 

7,388 

- 

- 



Sample size 

962 






SOURCES: MDRC calculations using data from the Family Rewards 42-month survey and New 
York State unemployment insurance (UI) wage records, and Seedco's Family Rewards program data. 


NOTES: Sample sizes may vary because of missing values. 
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Table 6.8 (continued) 


A two-tailed t-test was applied to differences between outcomes for the program and control groups. The 
p-value indicates the likelihood that the differences between the program and control groups arose by chance. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical significance levels 
(Sig.) are indicated as follows: fff = 1 percent; ff = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment 
characteristics of sample members. Standard errors were adjusted to account for multiple observations per 
family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Dollar averages include zero values for sample members who were not employed. 

UI records include only employment and earnings in jobs covered by the New York State U1 program. 
They do not include employment outside of New York State, nor in jobs not covered by the UI system (for 
example, "off-the -books" jobs and federal government jobs). 

A double dash (— ) indicates "not applicable." 


Employment effects by education level follow the same pattern, although the effects on 
UI employment are more negative for the less educated, with no effect on employment captured 
by the survey (as shown in Table 6.8). Although the earnings impact is not statistically different 
between the more and less educated subgroups, those without a high school diploma or General 
Educational Development (GED) certificate also saw a reduction of nearly $1,800 in UI wages 
over three years — about 9 percent of their total three-year earnings. The positive survey- 
reported employment effects are concentrated among parents who had a GED certificate or high 
school diploma at baseline. Those who were in extreme poverty when they entered the study 
also experienced negative UI employment effects similar to those who were less educated; the 
extreme poverty group, however, had increased survey employment rates that were comparable 
with the effects found for those with income at or above half of the federal poverty level at 
baseline (Appendix Table E.8). 

Consistent with the findings in the 2010 report, the program appears to have benefited 
those who were more work-ready at the time of random assignment by increasing survey- 
reported work that does not show up on UI records. For more disadvantaged subgroups, the 
program produced smaller effects on survey-reported employment and zero to negative effects 
on UI employment. 22 


~ 2 Appendix I in the supplement to this report presents differences in impacts on employment, education, 
and training for a number of exploratory subgroups; see Riccio et al. (2013), available at www.mdrc.org. 
Differences were explored between one- and two-parent families, foreign- and U.S.-bom families, parents who 
reported risk factors for depression and those who did not, food stamp recipients and nonrecipients, Temporary 
Assistance for Needy Families (TANF)/Safety Net Assistance (SNA) recipients and nonrecipients, families in 
different types of private or government-subsidized housing, part-time and full-time workers, and parents who 

(continued) 
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The “income effect” from the reward payments seems to be operating for these more 
disadvantaged groups. It makes sense that those who were least connected to the labor market 
and were least educated when they entered the study would be the quickest to reduce, or at least 
not increase, their work effort. During a worsening economy that likely made it more difficult 
for lower-skilled individuals to find work, the income effect may have been amplified. Appen- 
dix Tables E.2 and E.3 suggest that this may be the case, showing impacts that are less negative 
in Year 3, when the program was about to end and the economy was beginning to recover, than 
in Years 1 and 2. Since families were able to earn program rewards by completing education or 
health-related activities, parents who were less prepared for the labor market may have relied 
more on earning what they could from those two components of Family Rewards and less on 
looking for or maintaining full-time work. Some less work-ready individuals might have used 
the education and health rewards to cover various costs associated with acquiring more skills 
and credentials. The more disadvantaged groups did earn reward amounts that were a substan- 
tially larger share of their labor market earnings than did those who were more work-ready. 

Because families received substantial amounts of program income from the program’s 
comprehensive package of rewards, those who entered the study with lower job prospects and 
earnings stood to gain more as a share of their incomes by responding to these financial incen- 
tives than those who were more work-ready. For example, Table 6.8 shows that families who 
are led by adults with no high school-equivalent credential averaged over $6,300 in education 
and health rewards over three years, or about 30 percent of their earnings from Ul-covered jobs. 
For more educated adults, education and health rewards made up about 1 5 percent of their total 
UI earnings. The significant cash transfers that the more disadvantaged families received in the 
education and health domains suggests that more disadvantaged subgroups, who were likely 
less able to quickly enter the labor market, were much more likely to focus on the more attain- 
able education and health activities that the program rewarded and less on searching for jobs or 
increasing their work hours. 

As suggested above, some parents may have reduced their UI employment to obtain 
additional skills for the labor market. The employment effects described above are most 
pronounced for the education subgroups. Table 6.9 presents findings on education and training 
by education level. Those without a high school diploma or GED certificate may have addition- 
al barriers that make it difficult for them to persist in education and training. Similar to the 


self-reported different levels of health. The findings do not reveal many significant differences in subgroup 
impacts. Subgroups defined by a depression indicator and by food stamp receipt at random assignment showed 
the same pattern of negative UI employment and earnings impacts for the less work-ready subgroups as 
described in the text, with a large reduction in earnings for those at risk of depression and a small reduction in 
UI work for food stamp recipients. 
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The Opportunity NYC Demonstration: Family Rewards 

Table 6.9 


Impacts on Education and Training, by Respondent’s 
Education Level at the Time of Random Assignment 


Subgroup and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Sig. 

High school diploma/GED certificate 
or higher at baseline (%) 






Currently participating in any 
education or training activity 

14.4 

10.7 

3.8 ** 

0.017 


Has any trade license or training 
certification 

59.2 

52.2 

7 Q *** 

0.003 

t 

Has bachelor's degree or higher 

15.5 

13.9 

1.6 

0.340 


Sample size (total = 1,769) 

915 

854 




No high school diploma/GED certificate 
at baseline (%) 






Currently participating in any 
education or training activity 

11.4 

8.8 

2.7 

0.145 


Has any trade license or training 
certification 

47.3 

47.5 

-0.2 

0.944 

t 

Has bachelor's degree or higher 

0.7 

0.2 

0.5 

0.194 


Sample size (total = 1,128) 

592 

536 





SOURCE: MDRC calculations using data from the Family Rewards 42-month survey. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the differences between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical significance 
levels (Sig.) are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

This table reports on degrees and licenses received, regardless of whether they were received 
before or after random assignment. 
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employment findings, the program’s positive effects on education and training were concentrat- 
ed among the more highly educated group. When survey respondents were interviewed, 52 
percent of the more educated adults in the control group had a trade license or training certifica- 
tion, compared with 48 percent of the less-educated adults in the control group. The program 
increased the rate of holding a license or certificate by 7 percentage points among the more 
educated adults and did not affect those credentials for adults with less education. Education and 
training impacts for employment and income subgroups are shown in Appendix E. 23 

The program increased employment in non-UI-covered jobs and did not affect UI- 
covered employment for the full sample. The individuals who took non-UI-covered jobs were 
primarily service workers, and a large percentage of them worked in occupations related to 
child care. The prevalence of new child care and other service jobs among primarily single- 
parent households suggests that the kinds of non-UI jobs that less disadvantaged program 
members took may have been easier to access and provided more convenient options for them 
to take advantage of the full-time work rewards offer. 


Conclusion 

After three years of program operations, Family Rewards led to a small increase in college 
course participation and a small increase in getting a trade license or training certification. The 
program also increased employment, although most of that increase was in jobs that were not 
reported to the UI system. Survey-reported jobs that are not covered by UI data were heavily 
concentrated in service occupations, the majority of which were child care-related. Non-UI- 
covered jobs may not have offered as many benefits, but for low-income, single-parent house- 
holds, child care jobs may have been easier to find or more flexible in arranging work and 
family priorities. According to the survey, the program also increased earnings and benefits, 
indicating that although most of the new employment was not covered by UI data, some of 
these new jobs included better schedules and more benefits. 

The program’s positive employment impacts are more pronounced across a number of 
more work-ready subgroups. For adults who were more connected to or prepared for the labor 
market — meaning they entered the program with a high school-equivalent credential, were 
already employed, or had income in the past year at or above 50 percent of the federal poverty 
level — the program increased survey-reported work without affecting UI employment. 


~ 3 The program’s positive impacts on training were concentrated among those who were unemployed or 
living in severe poverty at baseline, and did not differ by income level. The employment subgroup finding 
suggests that individuals who were less connected to the labor market may have felt more motivated to acquire 
more skills, even though they did not necessarily reduce work to get more education or training. 
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In contrast, the program led to reductions in Ul-covered work for the less work-ready, 
with no or small effects on survey-reported employment. The reduction in Ul-covered work 
may be a response to the substantial rewards that these families were earning from the pro- 
gram’s health and education domains, or the program’s “income effect.” While not as strong as 
it could have been, the income effect that is apparent on the less work-ready subgroups, and not 
on the less disadvantaged groups, is a concern. Parents who were less work-ready were less 
connected to the labor market when the program ended, and longer-tenn follow-up will show 
whether these negative UI employment effects persist for them. 
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Chapter 7 

Conclusion 


When Family Rewards was launched in 2007, it sparked a great deal of interest in and specula- 
tion about how it would affect participating families. Conditional cash transfer (CCT) programs 
have reduced poverty and increased schooling and health care use in a range of low- and middle- 
income countries. But would a similar idea work in a very different context — a large city in a 
higher-income country? Since 2007, a lot has been learned about the feasibility of implementing 
a CCT program in a large, diverse city such as New York, what participating families think about 
the program, and its effects on their income, well-being, and other outcomes. 

Overall, the program’s effects were more modest than had been hoped. Its largest ef- 
fects were to reduce poverty and hardship during the three years that the cash rewards were 
offered. But the behavioral responses, the key to long-term poverty reduction, were either small 
or limited to subgroups. To be fair, the program led to some encouraging effects, such as the 
notable increase in high school graduation rates for ninth-graders who demonstrated proficiency 
in reading, or the big increase in dental care for all types of families. But it had no effects for 
younger students and small effects on other outcomes related to health care and parents’ 
employment. 

While the findings suggest that this version of a CCT program will not lead to big 
changes in long-term poverty, it is too early to make conclusions about the CCT model’s 
relevance for U.S. policy. Family Rewards represents just the first iteration of a CCT program 
in this country, and it led to enough positive changes to suggest that it could be made more 
effective if modified somewhat or targeted to certain types of families. Would the effects have 
been larger, for example, if the rewards were offered along with other services, if they were 
offered for somewhat different outcomes, or if they were targeted to specific families or 
students or to different geographic areas? It will be important to assess how adaptations to the 
model might make it more effective. The Great Recession highlighted the need for a safety net 
in the United States that provides low-income families with adequate support when jobs are 
hard to find, while also encouraging self-sufficiency. Continued experimentation with the CCT 
idea will be important to inform this debate. 

This chapter summarizes the findings to date from Family Rewards and places them in 
the context of findings from international CCT programs and incentives programs more 
generally in higher-income countries. It discusses what the findings suggest for the next version 
of a CCT program and describes this next version of Family Rewards, which is currently being 
tested in two cities in the United States. 
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Summary of Findings 

During the three years the program operated, the average participating family earned nearly 
$9,000 in rewards, or roughly $3,000 in each year. Nearly all families earned at least one 
education reward and one health reward during the program period, while just over 53 percent 
earned a work reward. As the program designers intended, families were able to and did earn 
rewards across a broad range of areas, and the significant amount that they earned increased 
their monthly incomes by 22 percent on average and reduced poverty and severe poverty by 
about 12 percentage points. However, the reductions in poverty and hardship began to diminish 
after Year 3, when the program ended. 

What were the effects on outcomes in the three areas in which rewards were offered? In 
terms of children’s school progress, Family Rewards improved outcomes for ninth-graders who 
were performing relatively better academically than their peers when they entered the study. For 
the reading-proficient group, although not for the math-proficient group, those early gains in 
performance led to sizable effects on grade promotion and on graduation. In contrast, the 
program had no effect for lower-performing ninth-graders or for elementary and middle school 
students. In the health area, early, positive effects on visits to the doctor and health status faded, 
although there were continued impacts on health coverage and, especially, dental visits. Finally, 
the program led to modest increases in employment throughout the follow-up period, although 
much of this new work was in jobs that were not reported to the unemployment insurance (UI) 
system. Also, the program may have led to a reduction in Ul-covered work for a subgroup of 
more disadvantaged adults in the study. 

The findings in this report are based on three to four years of follow-up data, covering 
up to one year after the program ended, but they are not the final word on its effects. First, some 
effects may emerge anew. In particular, a growing research documents the importance of family 
income and environment on the achievement of very young children, and many of the families 
in the study had children under the age of 6 when they first enrolled. A future report will 
examine whether the significant income boost that families received for three years has effects 
on these young children’s school progress. Second, it will be important to assess whether the 
positive effects observed to date persist beyond the program period. Given the large, positive 
effects for reading-proficient ninth-graders, for example, a future report will examine the effects 
on their college enrollment. 


Considerations for Designing Future Conditional Cash Transfer 
and Incentives Programs 

The Family Rewards findings add to a growing body of evidence on the effects of CCTs and 
more targeted incentives programs across a range of countries. This section considers the results 
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from Family Rewards and these other programs to discuss the implications of program design in 
each of the three domains that were studied — education, health, and work. 

The idea for Family Rewards came from CCT programs that have become so popular in 
low- and middle-income countries. One of the first programs, and the one that has been the 
most studied, is Mexico’s Oportunidades program, launched as Progresa in 1997. However, 
there are now CCT programs of varying scope and scale in most Latin American countries, as 
well as in several countries in Africa, the Middle East, and Asia. Most of the programs, which 
are often the main safety net program in the country, offer incentives for education (for school- 
age children) and health (typically for very young children), while a few target only education. 
Some are offered broadly to all low-income families in an area, while others are targeted to 
specific populations of concern. The programs overall have been found to reduce short-term 
poverty and increase consumption, particularly food consumption, with larger effects from 
programs that offer bigger transfers. 

A key question about the use of CCTs in a higher-income country such as the United 
States is whether they would work in a very different context. In lower-income countries, for 
example, the school attendance payment is designed to offset in part the opportunity costs (in 
terms of forgone income) that families face by keeping their children in school rather than 
sending them to work. 

Nonetheless, incentives have also become increasingly popular in the United States and 
other higher-income countries as a way to influence behavior. In education, however, they are 
less focused on school enrollment and more focused on performance. Examples of targeted 
behaviors include performance on standardized tests, taking and passing Advanced Placement 
courses, taking a full-time college course load, losing weight, quitting smoking, working full 
time, and participating in a training program. These programs tend to differ from Family 
Rewards in that they are not holistic but are targeted to one area or outcome. 

Education 

The CCT programs that have been run across a range of lower- and middle-income 
countries have generally led to increases in school attendance. 1 The effects have been larger at 
the secondary school level, since attendance at the primary (elementary school) level is already 
relatively high, and they have been larger in areas or countries where attendance rates would 
have been lowest in the absence of the program. In some cases the education impacts were also 
larger among the poorest households. 


'Fiszbein and Schady (2009). 
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However, few of these programs led to improvements in school performance. While 
none of them rewarded perfonnance per se, the expectation was that students would leam more 
if they attended more often. Some researchers posit that the lack of effects is a result of poor 
schools and poor home environments, neither of which was addressed by the CCTs. The 
exception to this finding is for very young children, where some studies suggest cognitive gains 
for children who were preschool-age when their families participated. 2 This finding is con- 
sistent with the increasingly popular idea that investments in early childhood can have im- 
portant, lasting effects. 3 

In the United States and other higher-income countries, a growing number of studies 
have tested financial incentives to increase student perfomiance. At the elementary and second- 
ary school levels, for example, incentives have been offered to improve reading time, test 
perfonnance, and course taking. 4 At the college level, there are programs to increase grades, 
courses taken, and persistence. 5 Studies that are particularly relevant to Family Rewards include 
tests of targeted incentives in four large U.S. school districts, including the SPARK program in 
New York City, 6 a study of incentives for test perfonnance in rural Ohio, 7 and a study of 
incentives given to Israeli high school students to take and pass high school exit exams. 8 

A general finding from these studies is that incentives can increase student effort, at 
least in the short run. 9 For example, the Israeli program encouraged more students to take and 
pass required high school exams, largely by encouraging some students to spend more time 
studying. However, it is not always clear that this increased effort while the incentives were 
offered translated into higher achievement in the long run. For example, a program that paid 
students in rural Ohio for perfomiance on annual, standardized tests found positive effects on 
math scores during the program period but no effects in later years when the rewards were not 
offered. Family Rewards’ effects on attendance and credits earned for proficient ninth-graders 
also diminished in Year 4, after the program had ended. 


2 Paxson and Schady (2008). 

Carneiro and Heckman (2003). 

4 Bettinger (2010); Jackson (2010); Fryer (2011). 

5 Patel and Rudd (2012); Leuven, Oosterbeek, and van der Klaauw (2010); Angrist, Lang, and Oreopou- 
los (2009). 

6 Fryer (2011). As mentioned in Chapter 1, SPARK was a school-based education incentives program de- 
signed to improve the school performance of fourth- and seventh-graders by rewarding good performance on a 
series of standardized tests that were administered over the course of the academic year. 

7 Bettinger (2010). 

s Angrist and Lavy (2009). 

q A substudy within Family Rewards also found, using a survey of ninth-graders, that the program in- 
creased proficient ninth-graders’ time spent in academic activities, suggesting the mechanism by which the 
program had its effects; see Morris, Aber, Wolf, and Berg (2012). 
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The design of the incentives also matters. More immediate incentives have larger ef- 
fects than rewards paid in the future; in-kind rewards can be as effective as monetary rewards 
for younger students; and rewards framed as losses (in which individuals are given a payment 
up front and must return some or all of it if they do not meet the required milestones) can have 
larger effects than those framed more traditionally as bonuses . 10 It has also been suggested, 
based on findings from the programs in four large schools districts, that incentives should 
reward inputs (for example, reading and homework), not outputs (such as test scores ). 11 Re- 
warding inputs may be more effective because students, and perhaps their parents, do not fully 
understand how to increase their achievement levels. Of course, this idea rests on the assump- 
tion that program designers know the right inputs to reward and can find a practical way to do 
so. Family Rewards did lead to some small increases in parents’ engagement with their chil- 
dren’s schooling, but those effects did not translate into improved school performance. 

Another finding from several of these studies, including Family Rewards, is that the in- 
centives had larger effects on students who were on the margins of higher perfomrance, or 
students who were within reach of the outcome that was rewarded . 12 For example, the positive 
effects on test scores for elementary school students in rural Ohio were larger for more academ- 
ically prepared students. The program that offered rewards for passing high school exit exams 
in Israel similarly had larger effects for more prepared students. Put another way, these are the 
students who could have performed relatively well, but would not have attended as much or 
fried as hard without the incentive . 13 

The findings from Family Rewards and these other studies suggest a number of consid- 
erations for the design of future incentives programs. First is the question of what behaviors to 
reward. Should the program reward inputs (such as attendance, homework, reading, test 
preparation, and tutoring) in addition to or instead of outputs (such as test scores and grades)? If 
students (and parents) would like to earn the rewards but do not know what steps to take to 
improve their school performance, rewarding inputs seems like an effective strategy. 

An alternative, or complement to this strategy, might be to offer more guidance to fami- 
lies about how to help their children perform better in school. The next version of Family 
Rewards includes more proactive guidance for families for identifying when their children are 
performing poorly and developing steps, such as tutoring, to help them improve their perfor- 
mance. Of course, an additional consideration here is the practicality of rewarding certain 


10 Levitt, List, Neckermann, and Sadoff(2012); Fryer, Levitt, List, and Sadoff(2012). 

"Fryer (2011). 

12 Bettinger (2010); Angrist and Lavy (2009); Leuven, Oosterbeek, and van der Klaauw (2010). 

13 Slavin (2010). In his review of incentives for school progress, Slavin questions the cost-effectiveness of 
these programs, arguing that other types of reforms, such as those that change an entire school, might affect 
more students than just those on the margin of success. 
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behaviors on a large scale. As an example, Family Rewards program designers considered 
offering rewards for completing homework. Given the wide variation in how schools and 
teachers assign homework and track its completion, however, implementing and verifying a 
compliance process was deemed impractical. 

On a broader level, it is worth considering whether the incentives should be combined 
with additional services, beyond staff guidance. Should they be combined with incentives for 
teachers, for example, which have been found to be effective in some circumstances ? 14 Should 
they be combined with broader school refonns, in an effort to improve the learning environment 
in which the students reside? One of the arguments made at the onset of Family Rewards was 
that the program did little to address the poor quality of some of the schools that these students 
attended . 15 While there is little evidence from Family Rewards that the incentives were more 
effective at higher-performing schools, this is an issue for further research. 

Finally, how often should the rewards be given and to whom should they be paid? The- 
ory suggests that more immediate incentives are more salient to individuals, and the recent 
research cited earlier confirms that student effort is less responsive to distant payments. In 
addition, in both Family Rewards and other programs, students often forgot about the incentives 
for certain activities until they received a payment. In Family Rewards, for example, payments 
for Regents exams passed or credits earned during the year were typically made during the 
summer. While a number of administrative constraints made it impossible to structure Family 
Rewards to provide immediate rewards for school achievement, the next version of the program 
incorporates several changes that move further in this direction. For example, it rewards 
performance more frequently by rewarding passing grades, paid at each report card period. In 
addition, payments for any rewards earned (in all three domains) are paid on a monthly basis, 
rather than every two months. 

Who should receive the rewards? Most of the targeted incentives that have been re- 
viewed provided payments directly to the students, even students as young as elementary 
school-age. Family Rewards, like most other CCT programs, gave rewards for elementary and 
middle school students to the parents and relied on them to tell their children about the incen- 
tives . 16 As a result, at least in Family Rewards, many of the younger children in the program 
were not fully aware that incentives were being offered. Programs might consider providing 
incentives directly to students or, at a minimum, providing more guidance to parents about how 
to tell their children about the incentives. The next version of Family Rewards targets only high 


14 Fryer, Levitt, List, and Sadoff(2012). 

15 The City attempted to address the issue of poor school quality through several other initiatives. 

16 This was done for practical reasons and because many parents would not have wanted their children to 
be given the sometimes large sums of money provided through the program. 
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school students for the education rewards, and they will receive the entire portion of each 
education incentive that they earn directly. 

Health 

As with effects on school attendance, most of the international CCT programs have 
been found to increase the use of health services . 17 Parents in the programs were more likely 
to have their children visit health clinics. Evidence on health outcomes, however, is more 
mixed. Some studies find increased height among the youngest children, while others find no 
effects on health status. 

In terms of more targeted programs in the United States, a growing research docu- 
ments that incentives can affect a variety of health-related behaviors, such as weight loss , 18 
smoking , 19 and adherence to prescribed medication . 20 In addition, a recent summary of studies 
found that the majority of incentives programs that were reviewed — providing rewards in 
the form of cash, lotteries, gifts, or coupons — were found to affect individuals’ behavior . 21 
Additionally, another review suggests that the payments need not be large to affect a variety 
of health-related outcomes . 22 

However, most of the U.S. studies are small, clinical trials, and most of them studied 
only behavior change in the short term. In the few studies that did track long-term outcomes, the 
effects typically faded after the program ended. Family Rewards’ modest effects on health care 
use and status followed a similar pattern. 

Most relevant to Family Rewards are incentives for more general outcomes, such as 
maintaining health insurance and visiting the doctor and dentist. Two studies in the review that is 
mentioned above did provide incentives for visiting the dentist, but there is little research to date 
on incentives to maintain health insurance, and recent research shows that having health insur- 
ance does increase the use of preventive care . 23 Family Rewards did lead to small increases in the 
rate of sustained health care coverage, but coverage rates in New York City were already much 
higher than anticipated . 24 While the program may have had much larger effects if tested in an 
area with low coverage, the Affordable Care Act may lessen the need for this particular reward. 


17 Fiszbein and Schady (2009); Lagarde, Haines, and Palmer (2007). 

18 Volpp et al. (2008a). 

19 Volpp et al. (2009). 

20 Volpp et al. (2008b). 

21 Kane, Johnson, Town, and Butler (2004). 

22 Sindelar (2008). 

23 Finkelstein et al. (2012). 

24 The Family Rewards sample does not include individuals without children or undocumented immi- 
grants, both groups with lower rates of health coverage. 
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However, the Act places significant emphasis on preventive care, through Medicaid pol- 
icies and other campaigns that are targeted to smoking, nutrition, exercise, and obesity. A holistic 
CCT program like Family Rewards may not be the best platform to tackle these types of behav- 
iors because it would likely require more intensive monitoring for compliance and verification, 
but it can work to address the first step in the process — encouraging individuals to make 
preventive care visits. (Although Family Rewards did increase preventive care dental visits, it did 
not increase the rate of preventive care doctors’ visits, which were already quite high. The 
program might have had an effect if it had been run in an area where such checkups are less 
common.) The findings from the existing research and Family Rewards suggest, however, that 
the incentives may need to be ongoing in order to affect behavior in the long term. 

Work 

To date, none of the major CCT programs in low- and middle-income counties has in- 
cluded rewards for parents’ work or training, although there was concern that the provision of 
education and health rewards may lead to a reduction in adults’ work effort. In general, evalua- 
tions have not found negative effects on work, with the exception of one program in Nicaragua, 
where the rewards for education and health were quite large. 25 The findings from Family 
Rewards also suggest this type of “income” effect, although the reduction in work was only for 
Ul-covered work and was found only for more disadvantaged adults in the study. 

The use of incentives to encourage work has a long history in the United States. The 
Earned Income Tax Credit (EITC) is the best and largest example of a program that increases 
the payoff to work, providing benefits to more than 27 million families. A range of studies 
suggests that expansions in the EITC significantly increased the employment rates of single 
mothers. 26 A number of recent evaluations have tested programs that provide monthly earnings 
supplements for full-time employment or rewards for sustained work. 27 In general, these 
programs do increase work, in some cases by moving more people into the labor market and in 
other cases by encouraging part-time workers to work full time. 

In several of the programs that were studied, 28 the effects on employment faded over 
time, even before the incentives offer had ended, meaning that the main effect of the incen- 
tives was to move individuals into work more quickly than they would have gone otherwise. 
However, for other groups, who would have had low employment rates in the absence of the 
program, the effects can last longer. Another finding from some studies is that incentives plus 

25 Fiszbein and Schady (2009). 

26 Holt (2006); Eissa and Hoynes (2006). 

27 Earlier studies are summarized in Michalopoulos (2005). More recent studies include Hendra et al. 
(2011) and Martinson and Hendra (2006). 

28 Michalopoulos (2005). 
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services are more effective at increasing employment than incentives alone, 29 suggesting that 
some people may need additional help moving into work to take advantage of the rewards. 
The next version of Family Rewards will include a more active advisement component, some 
of which might involve providing families with resources to help them find employment, if 
such assistance is desired. 

The evidence for training incentives is more mixed. Some recent programs did increase 
the take-up of training and the receipt of certificates. However, it is not clear that such training 
had much return in the labor market. In one program, there were no effects on employment, 
while in the other program the effects appear to have faded over time. 30 

This research has several implications for the design of work incentives in a CCT pro- 
gram. First, some families may choose to reduce their work effort in response to earning substan- 
tial rewards in the other areas. Chapter 6 presented evidence suggesting that this type of “income 
effect” occurred for more disadvantaged individuals in the study, but it is also possible that the 
rewards earned by individuals in the other two domains dampened the effectiveness of the work 
rewards more broadly. The answer is probably not to eliminate the work incentives altogether, 
since the reductions in work may have been bigger in the absence of those incentives. 

The work incentives might also be more effective if combined with employment ser- 
vices, ranging from basic job search preparation and assistance, to job development, to help 
with re-employment services if parents were to lose their jobs. 31 This type of guidance might 
make the work incentives more effective in general but might also help to stem the reduction in 
work that may occur among more disadvantaged individuals in response to the income that is 
available from the other rewards. 

Similarly, more guidance and direction are likely needed for the training incentives, to 
help direct individuals to the types of training that are both appropriate for them and relevant to 
the local labor market. The advisement assistance that was added to the model for the next 
version of Family Rewards also includes guidance for work and training, although perhaps not 
at the level of expertise that would be provided by a formal job developer. 

General Considerations 

Beyond the three domains, the research suggests additional issues for consideration. For 
example, should the CCT program be holistic? Family Rewards was deliberately designed to 
focus on three domains and to offer families many ways to earn rewards. This design helped to 


29 Michalopoulos (2005). 

30 Hendra et al. (2011); Miller, van Dok, Tessler, and Pennington (2012). 
31 Hamilton and Scrivener (2012). 
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ensure that all families could earn at least some rewards, and it undoubtedly led to the pro- 
gram’s substantial effects on income and poverty. However, it is possible that packaging the 
rewards together in this way may have reduced the power of any given incentive. With so many 
ways to earn rewards, for example, families could choose to focus on the “easy” ones and not 
stretch to meet the more challenging milestones. In addition, the large number of rewards, at 22, 
may have overwhelmed participants and led them to focus their efforts on just a few. 

Consider a program that offers only two incentives, one for high attendance at school 
and one for performance on standardized tests. Assuming for the moment that these are the right 
two behaviors to reward, will the program have more widespread effects on education if 
families have only those two rewards to focus on and to earn? Findings from at least some of 
the more targeted incentives programs suggest that it may have bigger effects. But the flip side 
of the coin is that such a pared-down program would not reduce poverty and material hardship 
nearly as much as Family Rewards did. With a greater number of incentives, families had a 
better chance to earn large cash transfers. 

One other design consideration relates to the target group and the role of CCTs in U.S. 
antipoverty policy. Some have suggested that a CCT program such as Family Rewards could be 
incorporated into or layered on top of a safety net program, such as Temporary Assistance for 
Needy Families (TANF). Politically, this program may be more popular and sustainable if it 
came with those types of strings attached. However, a nontrivial fraction of families did not earn 
very much from Family Rewards. The bottom fifth of earners received, on average, just over 
$700 per year from the program, while the top fifth earned $5,700 per year on average. The 
fonner group of families was a very disadvantaged subset of the study sample, with higher rates 
of unemployment, lower education levels, and higher rates of mental and physical health 
problems. Conditioning benefits on the achievement of certain milestones may put much of the 
safety net out of reach for the families who need it the most. 

On the other hand, as it stands, much of the current safety net is conditioned on only 
one activity — work (particularly, working in the case of the EITC and looking for work in 
the case of TANF). This system, the EITC in particular, worked well when jobs were 
plentiful but has proven inadequate in a bad labor market. 32 Conditioning benefit receipt on 
a range of productive activities (work, schooling, and health) may be a way to expand 
benefits to those in need. In addition, CCTs might be built into the current system in a way 
that provides families with enhanced payments or allows benefits to continue after they 
have reached a welfare time limit. 


32 Berlin (2011). 
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Finally, if the CCT idea were to play a larger role in U.S. policy, it is worth considering 
its costs. While there is no plan to conduct a formal benefit-cost analysis of Family Rewards, an 
analysis of the full costs of the program will be presented in a future report. The costs include 
the cash transfers to families, the costs of maintaining the payment system, and the costs of 
monitoring and enforcement for each of the reward conditions. The costs of Family Rewards 
are large and likely to outweigh the benefits, given the modest impacts it had in each of the 
domains. In addition, the monitoring and compliance costs are not trivial. While it is unlikely 
that an unconditional cash transfer to families would have led to similar effects in each of the 
domains, the lower cost would be one advantage of providing transfers with no strings attached. 
Political support for an unconditional cash transfer of this size, however, may be limited. 


Next Steps: “Family Rewards 2.0” 

The Center for Economic Opportunity and MDRC, with the Mayor’s Fund to Advance New 
York City, are replicating the Family Rewards model in two cities in the United States through 
grants provided by the federal Social Innovation Fund (SIF) and private funders. 33 In both New 
York City (the Bronx), and Memphis, Tennessee, just over 600 families were enrolled in each 
city’s program between September 2011 and February 2012. 34 Twice as many families were 
recruited for the study, with half randomly assigned to a program group, who are eligible for 
Family Rewards, and half assigned to a control group, who are not eligible for the program. 
With a sample of 1 ,200 families in each city, the evaluation will be able to examine program 
effects for the combined sample and for each city by itself. 

As mentioned throughout this chapter, the design of Family Rewards 2.0 builds on the 
lessons learned from this evaluation. For example, the new program targets families with 
children who were set to enter ninth or tenth grade in the upcoming school year, since effects on 
education were found in this study only for the older students. However, once enrolled in 
Family Rewards 2.0, all of the family’s school-age children are eligible for the health-related 
rewards. The program also targeted TANF and SNAP (food stamp) recipients, 35 in order to 
target resources to the neediest families and to consider how a CCT program might supplement 
or interact with these safety net programs. 

33 The SIF, enacted under the Edward M. Kennedy Serve America Act and administered by the Cor- 
poration for National and Community Service, targets public-private funds to expand and replicate 
proven approaches to pressing social problems through economic opportunity, healthy futures, youth 
development, and school support. See www.nationalservice.gov/about/programs/innovation.asp. 

34 Within New York City, the Bronx was selected for the replication because the original Family Rewards 
program was tested there, which would facilitate future comparisons between the original and new models, and 
also because among New York City’s low-income communities, those in the Bronx have among the highest 
poverty rates. 

35 SNAP is the federal Supplemental Nutrition Assistance Program, formerly the Food Stamp Program. 
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The new model also differs in several key ways from the original Family Rewards pro- 
gram. First, the new program is much simpler, offering only 8 incentives instead of 22, across 
the three domains. Second, the new program includes a more active advisement role for the 
staff. Recall that active case management was proscribed for staff in the original Family 
Rewards program, although they could certainly refer individuals to services if assistance was 
requested. While families generally reported satisfaction with the program, the implementation 
research indicated that many of them wanted to earn certain rewards but did not know what 
steps to take to meet the conditions. Those families might have benefited from more active 
guidance. Staff at the NPOs for Family Rewards 2.0 develop a Family Earnings Plan with every 
family and aim to meet with them twice a year to discuss their progress toward earning the 
rewards. Staff also use strategic outreach to engage families who are not earning rewards and 
have access to a small “resource fund” (discretionary money) to help families invest up front in 
services they may need to help them earn the rewards, such as short-term tutoring, work 
uniforms, transportation to job interviews, or certain licensing fees. Third, the new program 
attempts to make the rewards more timely (and thus more salient) by disbursing payments 
monthly, rather than every two months. Rewarding grades, in addition to test scores, also 
provides a more immediate incentive for school performance. 36 

Families will be eligible for the rewards for three years, and the evaluation will track 
their outcomes for five years to assess the effects of the rewards both during and after the 
program. The findings will help to assess whether the modifications to the model made it more 
effective for eligible families. In addition, the test of the program in Memphis as well as New 
York City will provide information on how the program’s effects might vary across different 
contexts within the United States. 


Conclusion 

A general idea behind CCTs and other incentives programs is that individuals sometimes do not 
make decisions in the short term that are in their longer-tenn interests. These types of behaviors 
run the gamut from not saving enough for retirement to eating foods that aren’t healthful, and 
occur across all types of individuals. The reasons may vary and are often linked to basic human 
nature, such as the tendency to maintain the status quo, for example, or to succumb to peer 
pressure. Other reasons include a lack of willpower, discounting the future too heavily, a lack of 


36 The program also tries to tap into the power of “loss aversion." (Recall that incentives framed as losses 
have been found to be more effective than those framed as gains.) Each pay period, families will receive a 
statement indicating what they did earn and the amount they could have earned had they met additional 
milestones. 
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information, or a lack of resources. This recognition has given rise to an increasing use of 
incentives and other nonmonetary “nudges” to help individuals make better choices. 37 

Poor families are especially challenged in their ability to make decisions that serve their 
longer-tenn interests, given that they must devote significant time and energy to manage in the 
short term with limited resources . 38 Family Rewards was the first comprehensive CCT program 
in the United States that attempted to help and encourage families to make more investments in 
their human capital, while at the same time reducing their current levels of poverty and material 
hardship. The findings presented in this report show that in some cases the incentives worked 
and in other cases they did not. These findings add to the growing body of evidence on the 
promise and challenges of using incentives to influence behavior. They have also helped to 
shape a revised Family Rewards model and led to a test of this revised model in a new city. 
Future reports on the original and new models will provide more evidence on the potential value 
of a CCT approach in the United States. 


37 Thaler and Sunstein (2008). 
38 Mullainathan and Shafir (2012). 
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The Opportunity NYC Demonstration: Family Rewards 
Appendix Table A.l 

Data Sources and Coverage Periods for Current Report 


Data Source 

Months for Which 
Data Were Collected 

Length of Follow-Up Period 
for Quantitative Measures 

End of Follow-Up Period Relative 
to Program Y ears 1 and 3 

Unemployment insurance wage 
records 3 

July 2007 - September 2010 

36 months from each family's random 
assignment date 

Near end of third program year for 
most sample members 

Temporary Assistance for 
Needy Families/Safety Net 
Assistance records 

July 2007 - January 2011 

36 months from each family's random 
assignment date 

End of third program year for all 
sample members 

Food stamp records 

July 2007 - January 2011 

36 months from each family's random 
assignment date 

End of third program year for all 
sample members 

Medicaid records’ 3 

July 2007 - January 2011 

36 months from each family's random 
assignment date 

End of third program year for all 
sample members 

School records 

September 2007 - August 2011 

1 year after end of the program 

1 year after end of the program for 
all sample members 

Family Rewards data from 
Seedco c 

September 2007 - July 2011 

3 complete program years 

End of third program year for most 
sample members 

Survey 

November 2010 - June 2011 

42 months (average) from each 
family's random assignment date 

End of third program year for all 
sample members 

Program observations and 
in-depth interviews 

July 2007 - March 2011 

— 

— 


NOTES: A double dash (--) indicates "not applicable." 

“Unemployment insurance (UI) wage records are provided in calendar quarters (quarter 3, 2007, through quarter 3, 2010). 
b Medicaid receipt was measured by collecting snapshots of the sample's Medicaid status during the first day of each calendar quarter. 
“This source refers to the data that Seedco collects on the rewards that program participants earned. These data do not include 
information on payments made to participants. 



The Opportunity NYC Demonstration: Family Rewards 
Appendix Table A.2 

Characteristics of Families at the Time of Random Assignment 

Characteristic Total 

One-parent family 3 (%) 80.9 

Two-parent family with both parents enrolled in Family Rewards study b (%) 5.7 

Number of children in household (%) 

1 child 22.8 

2 children 34.2 

3 children 25.1 

4 children or more 17.8 

Average number of children in household 2.5 

Primary language spoken at home is Spanish (%) 21.7 

Housing status (%) 

Own home or apartment 5.9 

Rent home or apartment 87.1 

Other housing arrangement 7.1 

Living in public housing (%) 30.3 

Receiving Section 8 rental assistance (%) 23.0 

Receiving TANF or Safety Net Assistance 0 (%) 24.0 

Receiving food stamps (%) 59.4 

At least one adult covered by public health insurance (%) 70.9 

Not receiving any public benefits or housing assistance d (%) 13.1 

Earnings above 130% of federal poverty level 0 (%) 14.9 

Sample size 4,749 

SOURCE: MDRC calculations using data from Baseline Information Forms. 

NOTES: In order to assess differences in characteristics across research groups, chi-square tests were used 
for categorical variables, and t-tests were used for continuous variables. 

Sample sizes may vary because of missing values. 

Rounding may cause slight discrepancies in calculating sums. 

Public health insurance measures in this table exclude child information. 

a This measure includes families with parents who reported their marital status as single, single but 
living with a boyfriend or girlfriend, separated, divorced, or widowed. 

b This measure refers to sample members who enrolled in the Family Rewards study with their spouse 
or legal domestic partner. 

c This measure includes families with child-only cases. 

d This measure refers to families who were not receiving TANF/Safety Net Assistance, food stamps, 
Medicaid, or Section 8 housing vouchers and did not live in public housing. 

c Income amounts from sources other than earnings were not available from the Baseline Information 
Form. 
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The Opportunity NYC Demonstration: Family Rewards 
Appendix Table A.3 


Characteristics of Parents at the Time of Random Assignment 


Characteristic 

Total 

Female (%) 

94.3 

Age (%) 

18-24 years 

0.3 

25-34 years 

28.9 

35-44 years 

45.2 

45-59 years 

22.8 

60 years or more 

2.9 

Average age (years) 

39.9 

U.S. citizen 3 (%) 

83.1 

By birth 

67.4 

By naturalization 

15.7 

Legal Permanent Resident (%) 

16.9 

Race/ethnicity (%) 

Hispanic/Latino 

46.7 

White, non-Hispanic/Latino 

0.7 

Black, non-Hispanic/Latino 

51.2 

Other 

1.4 

Education (highest degree or diploma earned) b (%) 

GED certificate 

8.9 

High school diploma 

17.1 

Some college 

19.7 

Associate's degree/2-year college 

7.1 

4-year college or beyond 

6.9 

None of the above 

40.2 

Marital status (%) 

Single 

62.5 

Cohabitating 

2.3 

Separated, widowed, or divorced 

16.1 

Married or in a legal domestic partnership 

19.1 

Has an account at bank or credit union (%) 

47.0 

Employment measures 

Currently working (%) 

53.0 

Working full time 1 (%) 

40.2 

Average weekly earnings, among those currently working ($) 

394 

During past year, average number of months worked d 

9.9 


(continued) 
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Appendix Table A.3 (continued) 


Characteristic 

Total 

Health measures (%) 


Health insurance coverage 


Public health insurance 

70.5 

Employer health insurance 

20.6 

Other health insurance 

3.0 

Not covered 

6.0 

Had annual medical checkup when not sick 


Within the past year 

82.1 

1-2 years ago 

14.3 

More than 2 years ago 

3.4 

Never 

0.2 

Last medical checkup was at own (regular) doctor's 
office or clinic 

95.1 

Had preventive dental checkup 


Within the past year 

64.9 

1-2 years ago 

23.5 

More than 2 years ago 

10.9 

Never 

0.7 

Self-rated health 


Excellent or very good 

43.0 

Good 

36.9 

Fair or poor 

20.1 

Over the past 2 weeks, 


Had little or no interest in doing things 

22.4 

Had been feeling down, depressed, or hopeless 
Had little or no interest in doing things, and 

21.9 

had been feeling down, depressed, or hopeless 

13.8 

Sample size 

4,749 


SOURCE: MDRC calculations using data from Baseline Information Forms. 


NOTES: In order to assess differences in characteristics across research groups, chi-square tests were 
used for categorical variables, and t-tests were used for continuous variables. 

Sample sizes may vary because of missing values. 

Rounding may cause slight discrepancies in calculating sums. 

This table excludes information for enrolled second parents in two-parent households (N = 247). 
a This measure refers to U.S. citizens both by birth and by naturalization. 
b ln the earlier report on Family Rewards (Riccio et al., 2010), individuals who reported “some 
college” were excluded from the educational attainment measure; thus, the percentages shown here 
differ slightly from the percentages shown for educational attainment in the earlier report. 

c This measure refers to 30 hours a week or more. In the earlier report on Family Rewards (Riccio 
et al., 2010), some responses were inadvertently omitted from the "working full time" measure; the 
percentage shown here is the correct one. 

d This measure shows the number of months individuals worked during the past year, regardless of 
employment status at the time of random assignment. 
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The Opportunity NYC Demonstration: Family Rewards 
Appendix Table A.4 


Characteristics of All Children at the 
Time of Random Assignment 


Characteristic 

Total 

Gender (%) 

Female 

50.0 

Male 

50.0 

Age (%) 

0-5 years 

13.7 

6-10 years 

30.6 

1 1-13 years 

26.2 

14 years or older 

29.5 

Born in the United States (%) 

93.5 

Race/ethnicity (%) 

Hispanic/Latino 

47.0 

White, non-Hispanic/Latino 

0.4 

Black, non-Hispanic/Latino 

50.4 

Other 

2.2 

Type of school child attended in the past year (%) 

Public or charter school 

97.5 

Private or parochial school 

2.5 

Grade" (%) 

Not yet in pre-K or kindergarten 

7.1 

Pre-K 

2.4 

Kindergarten 

2.8 

Grade 1 

3.5 

Grade 2 

4.2 

Grade 3 

3.8 

Grade 4 

16.7 

Grade 5 

3.9 

Grade 6 

4.3 

Grade 7 

16.1 

Grade 8 

3.9 

Grade 9 

18.6 

Grade 10 

4.1 

Grade 1 1 

3.6 

Grade 12 

3.0 

College 15 

1.2 

Not enrolled" 

0.9 


(continued) 
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Appendix Table A.4 (continued) 


Characteristic 

Total 

Education measures (%) 


Child's parent attended parent-teacher conference 
during past year 


Never 

5.0 

1-2 times 

35.3 

3-4 times 

35.0 

5-6 times 

11.8 

More than 6 times 

12.9 

Child's parent spoke with teacher about 
grades, tests, or homework during past year 


Not at all 

2.9 

A little 

8.2 

Some 

20.0 

A lot 

34.5 

A great deal 

34.5 

Enrolled in special education in the past school year 

13.5 

Enrolled as an English Language Learner 
in the past school year 

12.9 

Child health outcomes (%) 


Health insurance coverage 


Public health insurance 

81.1 

Employer health insurance 

14.5 

Other health insurance 

1.7 

Not covered 

2.7 

Parent's rating of child's health 


Excellent 

43.5 

Very good 

31.1 

Good 

21.8 

Fair 

3.1 

Poor 

0.4 

Had annual medical checkup when not sick 


Within the past year 

90.8 

1-2 years ago 

8.3 

More than 2 years ago 

0.8 

Never 

0.2 

Last annual checkup was at own (regular) doctor's office or clinic 

97.7 

(continued) 
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Appendix Table A.4 (continued) 


Characteristic 

Total 

Flad preventive dental checkup 


Within the past year 

74.6 

1-2 years ago 

17.2 

More than 2 years ago 

3.1 

Never 

5.1 

Idas a physical problem that limits activities 

9.5 

Has an emotional or mental health problem 
that limits activities 

6.3 

Has a physical, emotional, or mental health problem 
that limits activities 

13.3 

Sample size 

11,329 


SOURCE: MDRC calculations using data from Baseline Information Forms. 

NOTES: In order to assess differences in characteristics across research groups, chi-square 
tests were used for categorical variables, and t-tests were used for continuous variables. 
Sample sizes may vary because of missing values. 

Rounding may cause slight discrepancies in calculating sums. 

Children whose parents withdrew from the study were excluded from the sample, 
resulting in slight discrepancies on some measures between this report and the earlier report 
on Family Rewards (Riccio et al., 2010). 

“Grades 4, 7, and 9 were "target grades" for the Family Rewards program. Therefore, 
all enrolled families had to have a child in grade 4, 7, or 9. 

b College students who were under the age of 18 were enrolled in Family Rewards. 

They were not eligible for any reward payments. 

c The "not enrolled" category includes school-age children who are no longer attending 
or have graduated before the age of 18. 
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The Opportunity NYC Demonstration: Family Rewards 
Appendix Table B.l 


Proportion of Annual Reward Earnings per Family Made Up of 
Education Rewards Earned by Students in Grade 9 at the Time of 
Random Assignment, by English Language Arts (ELA) Proficiency 


Sample and Program Year 

Average T otal Reward 
Earnings per Family ($) 

Education Reward 
Earnings (%) 

Proficient on 8th grade ELA test 3 

Year 1 

4,422 

39.5 

Year 2 

4,418 

38.7 

Year 3 

3,600 

49.0 

Years 1, 2, and 3 combined 

12,291 

42.9 

Sample size 


272 

Not proficient on 8th grade ELA test 3 

Year 1 

3,279 

29.6 

Year 2 

3,185 

28.0 

Year 3 

2,809 

41.8 

Years 1, 2, and 3 combined 

8,946 

32.6 

Sample size 


553 


SOURCE: MDRC calculations using Seedco's Family Rewards program data. 


NOTE: Calculations are based on families who earned any rewards. 

Education reward earnings are presented as a percentage of total reward earnings per family. 
a ln New York State, students who score at a level of 3 or higher on a 4-point scale are deemed 
"proficient." 
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The Opportunity NYC Demonstration: Family Rewards 


Appendix Table C.l 

Impacts on School Outcomes for Students in Grade 4 at the 
Time of Random Assignment, by Performance in the Prior Year (Grade 3) 


Subgroup and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value Sig. 

Proficient on 3rd urade math test a (%) 





Attendance rate, Years 1 to 3 

92.3 

91.9 

0.4 

0.526 

Attendance rate, Year 4 

83.2 

83.2 

-0.1 

0.976 

Scored at proficient level or higher on ELA, Year 4 

29.9 

31.7 

-1.8 

0.455 

Scored at proficient level or higher on math, Year 4 

53.2 

53.1 

0.1 

0.972 

Sample size (total = 1,290) 

656 

634 



Not proficient on 3rd erade math test a (%) 





Attendance rate, Years 1 to 3 

89.5 

90.4 

-0.9 

0.370 

Attendance rate, Year 4 

81.9 

81.4 

0.6 

0.846 

Scored at proficient level or higher on ELA, Year 4 

7.3 

4.4 

3.0 

0.323 

Scored at proficient level or higher on math, Year 4 

13.1 

9.2 

3.9 

0.288 

Sample size (total = 377) 

178 

199 



Proficient on 3rd srade ELA test a (%) 





Attendance rate, Years 1 to 3 

92.1 

92.6 

-0.5 

0.505 

Attendance rate, Year 4 

81.4 

82.4 

-1.0 

0.672 

Scored at proficient level or higher on ELA, Year 4 

41.9 

46.3 

-4.3 

0.238 

Scored at proficient level or higher on math, Year 4 

59.0 

63.9 

-4.9 

0.167 ft 

Sample size (total = 760) 

382 

378 




(continued) 
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Appendix Table C.l (continued) 


Subgroup and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Sig. 

Not proficient on 3rd srade ELA test a (%) 

Attendance rate, Years 1 to 3 

91.2 

90.5 

0.7 

0.371 


Attendance rate, Year 4 

84.8 

82.9 

1.9 

0.296 


Scored at proficient level or higher on ELA, Year 4 

10.6 

8.5 

2.0 

0.337 


Scored at proficient level or higher on math, Year 4 

32.1 

25.5 

6.6 ** 

0.030 

tt 

Sample size (total = 878) 

437 

441 





SOURCE: MDRC calculations using data from New York City Department of Education administrative 
records. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the difference between the program and control groups 
arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 
1 0 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical significance 
levels (Sig.) are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. Standard errors were adjusted to account for multiple 
observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Years 1, 2, 3, and 4 cover the 2007-2008, 2008-2009, 2009-2010, and 2010-201 1 school years, 
respectively. 

ELA = English language arts. 

a In New York State, students who score at a level of 3 or higher on a 4-point scale are deemed 
"proficient." 


218 






The Opportunity NYC Demonstration: Family Rewards 
Appendix Table C.2 

Impacts on School Outcomes for Students in Grade 4 at the 
Time of Random Assignment, by Parents’ Education Level 



Program 

Control 

Difference 


Subgroup and Outcome 

Group 

Group 

(Impact) 

P- Value 

Parent has high school diploma/GED 
certificate or higher at baseline (%) 





Attendance rate, Y ears 1 to 3 

92.2 

92.1 

0.1 

0.822 

Attendance rate, Y ear 4 

82.5 

81.9 

0.6 

0.758 

Scored at proficient level or higher on ELA, 
Year 4 a 

Scored at proficient level or higher on math. 

27.9 

28.0 

-0.1 

0.974 

Year 4 a 

47.9 

46.6 

1.3 

0.656 

Sample size (total = 1,035) 

495 

540 



Parent has no high school diploma/GED 
certificate at baseline (%) 





Attendance rate, Years 1 to 3 

90.3 

90.0 

0.4 

0.702 

Attendance rate, Y ear 4 

82.0 

83.4 

-1.4 

0.503 

Scored at proficient level or higher on ELA, 
Year 4 a 

Scored at proficient level or higher on math. 

19.1 

19.7 

-0.6 

0.854 

Year 4 a 

38.0 

37.6 

0.5 

0.898 

Sample size (total = 638) 

342 

296 




SOURCE: MDRC calculations using data from New York City Department of Education 
administrative records. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and 
control groups. The p-value indicates the likelihood that the difference between the program and 
control groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 
percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. No statistically 
significant differences between subgroup impacts were observed. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. Standard errors were adjusted to account for 
multiple observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Years 1, 2, 3, and 4 cover the 2007-2008, 2008-2009, 2009-2010, and 2010-2011 school years, 
respectively. 

ELA = English language arts. 

a ln New York State, students who score at a level of 3 or higher on a 4-point scale are deemed 
"proficient." 
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The Opportunity NYC Demonstration: Family Rewards 
Appendix Table C.3 

Impacts on School Outcomes for Students in Grade 4 at the 
Time of Random Assignment, by School Environment 



Program 

Control 

Difference 



Subgroup and Outcome 

Group 

Group 

(Impact) 

P- Value 

Sig. 

Students in lower-ranking schools 
at baseline (%) 






Attendance rate, Years 1 to 3 

89.4 

91.8 

-2.4 ** 

0.023 

tft 

Attendance rate, Y ear 4 

79.9 

85.5 

-5.5 ** 

0.025 

ft 

Scored at proficient level or higher on ELA, 
Year 4 a 

Scored at proficient level or higher on math. 

21.2 

23.2 

-2.0 

0.571 


Year 4 a 

44.7 

39.1 

5.6 

0.170 


Sample size (total = 531) 

255 

276 




Students in medium-ranking schools 
at baseline (%) 






Attendance rate, Y ears 1 to 3 

92.4 

91.3 

1.1 

0.197 

fff 

Attendance rate, Y ear 4 

84.3 

81.7 

2.6 

0.240 

ft 

Scored at proficient level or higher on ELA, 
Year 4 a 

Scored at proficient level or higher on math. 

23.3 

18.8 

4.6 

0.152 


Year 4 a 

41.7 

43.2 

-1.5 

0.688 


Sample size (total = 664) 

326 

338 




Students in higher-ranking schools 
at baseline (%) 






Attendance rate, Years 1 to 3 

93.3 

91.4 

J C) ** 

0.028 

fff 

Attendance rate, Y ear 4 

84.7 

81.3 

3.4 

0.239 

ff 

Scored at proficient level or higher on ELA, 
Year 4 a 

Scored at proficient level or higher on math. 

32.9 

35.1 

-2.2 

0.610 


Year 4 a 

50.6 

48.0 

2.6 

0.569 


Sample size (total = 438) 

226 

212 
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Appendix Table C.3 (continued) 


SOURCE: MDRC calculations using data from New York City Department of Education 
administrative records. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the difference between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** 

= 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical 
significance levels (Sig.) are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. Standard errors were adjusted to account for multiple 
observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Years 1, 2, 3, and 4 cover the 2007-2008, 2008-2009, 2009-2010, and 2010-2011 school years, 
respectively. 

ELA = English language arts. 

School environment was defined using ELA and math scores for all students in grade 4 
attending a school in the two years prior to the study. Schools were then divided into thirds based on 
the ranking (lower, medium, and higher) of their average scores. 

a ln New York State, students who score at a level of 3 or higher on a 4-point scale are deemed 
"proficient." 
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The Opportunity NYC Demonstration: Family Rewards 
Appendix Table C.4 

Impacts on School Outcomes for Students in Grade 7 at the 
Time of Random Assignment, by Performance in the Prior Year (Grade 6) 


Subgroup and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Proficient on 6th grade math test a (%) 





Attendance rate, Years 1 to 3 

91.6 

91.5 

0.2 

0.794 

Attendance rate. Year 4 

76.9 

78.6 

-1.7 

0.415 

Earned at least 1 1 credits, Year 4 

54.7 

59.1 

-4.4 

0.190 

Sample size (total = 854) 

432 

422 



Not proficient on 6th grade math test 3 (%) 





Attendance rate, Years 1 to 3 

86.3 

86.4 

-0.1 

0.928 

Attendance rate, Y ear 4 

68.8 

70.3 

-1.4 

0.532 

Earned at least 1 1 credits, Year 4 

38.8 

37.7 

1.1 

0.767 

Sample size (total = 761) 

370 

391 



Proficient on 6th grade ELA test 3 (%) 





Attendance rate, Years 1 to 3 

91.9 

91.1 

0.8 

0.272 

Attendance rate, Y ear 4 

76.0 

76.8 

-0.8 

0.762 

Earned at least 1 1 credits, Year 4 

54.0 

54.6 

-0.6 

0.877 

Sample size (total = 600) 

301 

299 



Not proficient on 6th grade ELA test 3 (%) 





Attendance rate, Years 1 to 3 

87.6 

87.7 

-0.1 

0.903 

Attendance rate, Y ear 4 

71.5 

73.0 

-1.5 

0.455 

Earned at least 1 1 credits, Year 4 

43.0 

45.6 

-2.5 

0.415 

Sample size (total = 993) 

490 

503 
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Appendix Table C.4 (continued) 


SOURCE: MDRC calculations using data from New York City Department of Education 
administrative records. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the difference between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 
5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. No statistically 
significant differences between subgroup impacts were observed. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. Standard errors were adjusted to account for multiple 
observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Years 1, 2, 3, and 4 cover the 2007-2008, 2008-2009, 2009-2010, and 2010-2011 school years, 
respectively. 

ELA = English language arts. 

a ln New York State, students who score at a level of 3 or higher on a 4-point scale are deemed 
"proficient." 
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The Opportunity NYC Demonstration: Family Rewards 


Appendix Table C.5 

Impacts on School Outcomes for Students in Grade 7 at the 
Time of Random Assignment, by Parents’ Education Level 



Program 

Control 

Difference 


Subgroup and Outcome 

Group 

Group 

(Impact) 

P- Value 

Parent has high school diploma/GED 
certificate or higher at baseline (%) 





Attendance rate, Y ears 1 to 3 

90.3 

89.6 

0.7 

0.312 

Attendance rate, Y ear 4 

75.0 

74.9 

0.2 

0.934 

Earned at least 1 1 credits, Year 4 

49.7 

50.5 

-0.8 

0.812 

Sample size (total = 932) 

472 

460 



Parent has no high school diploma/GED 
certificate at baseline (%) 





Attendance rate, Y ears 1 to 3 

87.6 

88.0 

-0.4 

0.653 

Attendance rate, Y ear 4 

69.9 

73.4 

-3.5 

0.129 

Earned at least 1 1 credits, Y ear 4 

41.8 

44.5 

-2.7 

0.475 

Sample size (total = 680) 

325 

355 




SOURCE: MDRC calculations using data from New York City Department of Education 
administrative records. 


NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and 
control groups. The p-value indicates the likelihood that the difference between the program 
and control groups arose by chance. Statistical significance levels are indicated as follows: *** 
= 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. No statistically 
significant differences between subgroup impacts were observed. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre- 
random assignment characteristics of sample members. Standard errors were adjusted to 
account for multiple observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Years 1, 2, 3, and 4 cover the 2007-2008, 2008-2009, 2009-2010, and 2010-2011 school 
years, respectively. 
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The Opportunity NYC Demonstration: Family Rewards 
Appendix Table C.6 

Impacts on School Outcomes for Students in Grade 7 at the 
Time of Random Assignment, by School Environment 



Program 

Control 

Difference 



Subgroup and Outcome 

Group 

Group 

(Impact) 

P-Value Sig. 

Students in lower-ranking schools 
at baseline (%) 






Attendance rate, Y ears 1 to 3 

86.3 

87.0 

-0.7 

0.503 

tt 

Attendance rate, Y ear 4 

69.5 

73.0 

-3.5 

0.187 


Earned at least 1 1 credits, Year 4 

40.8 

45.1 

-4.2 

0.317 


Sample size (total = 562) 

266 

296 




Students in medium-ranking schools 
at baseline (%) 






Attendance rate, Years 1 to 3 

88.4 

89.3 

-0.9 

0.374 

ft 

Attendance rate, Y ear 4 

73.2 

74.3 

-1.1 

0.716 


Earned at least 1 1 credits, Year 4 

50.3 

50.4 

0.0 

0.994 


Sample size (total = 453) 

213 

240 




Students in higher-ranking schools 
at baseline (%) 






Attendance rate, Years 1 to 3 

92.3 

90.0 

2 3 *** 

0.009 

ft 

Attendance rate, Y ear 4 

76.8 

74.9 

1.9 

0.518 


Earned at least 1 1 credits, Year 4 

57.4 

60.0 

-2.6 

0.580 


Sample size (total = 457) 

241 

216 
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Appendix Table C.6 (continued) 

SOURCE: MDRC calculations using data from New York City Department of Education 
administrative records. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and 
control groups. The p-value indicates the likelihood that the difference between the program 
and control groups arose by chance. Statistical significance levels are indicated as follows: *** 
= 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical 
significance levels (Sig.) are indicated as follows: fff = 1 percent; f | = 5 percent; f = 10 
percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre- 
random assignment characteristics of sample members. Standard errors were adjusted to 
account for multiple observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Years 1, 2, 3, and 4 cover the 2007-2008, 2008-2009, 2009-2010, and 2010-2011 school 
years, respectively. 

School environment was defined using ELA and math scores for all students in grade 7 
attending a school in the two years prior to the study. Schools were then divided into thirds 
based on the ranking (lower, medium, and higher) of their average scores. 
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The Opportunity NYC Demonstration: Family Rewards 
Appendix Table C.7 

Impacts on School Outcomes for Students in Grade 9 at the 
Time of Random Assignment, by Parents’ Education Level 



Program 

Control 

Difference 


Subgroup and Outcome 

Group 

Group 

(Impact) 

P- Value 

Parent has high school diploma/GED 
certificate or higher at baseline (%) 





Graduated within 4 years (%) 

53.7 

53.6 

0.1 

0.967 

Attendance rate, Y ears 1 to 3 (%) 

80.4 

79.6 

0.8 

0.523 

Attendance rate, Y ear 4 (%) 

64.0 

62.5 

1.5 

0.457 

Earned at least 44 credits, Y ears 1 to 4 (%) 

45.0 

43.9 

1.1 

0.680 

Average number of credits earned, Years 1 to 4 

34.6 

33.7 

0.8 

0.408 

Passed at least 5 Regents exams, Years 1 to 4 (%) 

41.1 

39.3 

1.8 

0.474 

Sample size (total = 1,166) 

559 

607 



Parent has no high school diploma/GED 
certificate at baseline (%) 





Graduated within 4 years (%) 

41.2 

40.7 

0.5 

0.884 

Attendance rate, Years 1 to 3 (%) 

75.7 

75.2 

0.6 

0.745 

Attendance rate, Y ear 4 (%) 

55.8 

57.1 

-1.3 

0.644 

Earned at least 44 credits, Years 1 to 4 (%) 

35.4 

35.6 

-0.2 

0.963 

Average number of credits earned, Years 1 to 4 

29.6 

29.3 

0.3 

0.846 

Passed at least 5 Regents exams, Years 1 to 4 (%) 

30.4 

31.8 

-1.4 

0.645 

Sample size (total = 744) 

392 

352 
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Appendix Table C.7 (continued) 

SOURCE: MDRC calculations using data from New York City Department of Education 
administrative records. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the difference between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 
5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. No statistically 
significant differences between subgroup impacts were observed. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. Standard errors were adjusted to account for multiple 
observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Years 1, 2, 3, and 4 cover the 2007-2008, 2008-2009, 2009-2010, and 2010-2011 school years, 
respectively. 

The Regents measures in this table include the following Regents exams: English, Math A, Math 
B, Geometry, Integrated Algebra, Algebra 2/Trigonometry, U.S. History and Government, Global 
History and Geography, Living Environment, Chemistry, Physics, and Earth Science. 
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The Opportunity NYC Demonstration: Family Rewards 
Appendix Table C.8 

Impacts on School Outcomes for Students in Grade 9 at the 
Time of Random Assignment, by School Environment 


Subgroup and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Students in lower-ranking schools at baseline 





Graduated within 4 years (%) 

38.6 

37.7 

0.9 

0.792 

Attendance rate, Years 1 to 3 (%) 

73.1 

73.8 

-0.7 

0.689 

Attendance rate, Year 4 (%) 

53.5 

52.7 

0.8 

0.767 

Earned at least 44 credits, Years 1 to 4 (%) 

33.2 

32.9 

0.4 

0.914 

Average number of credits earned, Years 1 to 4 

29.1 

28.7 

0.4 

0.754 

Passed at least 5 Regents exams, Y ears 1 to 4 (%) 

27.2 

25.8 

1.4 

0.645 

Sample size (total = 745) 

342 

403 



Students in medium-ranking schools at baseline 





Graduated within 4 years (%) 

53.3 

49.3 

4.0 

0.342 

Attendance rate, Years 1 to 3 (%) 

80.6 

79.4 

1.2 

0.532 

Attendance rate, Y ear 4 (%) 

63.8 

62.0 

1.8 

0.560 

Earned at least 44 credits, Y ears 1 to 4 (%) 

45.6 

43.3 

2.3 

0.584 

Average number of credits earned, Years 1 to 4 

34.3 

33.8 

0.5 

0.743 

Passed at least 5 Regents exams, Years 1 to 4 (%) 

38.6 

34.4 

4.2 

0.271 

Sample size (total = 555) 

287 

268 



Students in higher-ranking schools at baseline 





Graduated within 4 years (%) 

63.6 

68.2 

-4.5 

0.280 

Attendance rate, Years 1 to 3 (%) 

85.6 

83.4 

2.2 

0.156 

Attendance rate, Year 4 (%) 

70.0 

68.3 

1.8 

0.530 

Earned at least 44 credits, Y ears 1 to 4 (%) 

53.1 

52.6 

0.5 

0.914 

Average number of credits earned, Years 1 to 4 

39.4 

37.8 

1.6 

0.286 

Passed at least 5 Regents exams, Years 1 to 4 (%) 

54.2 

56.0 

-1.8 

0.656 

Sample size (total = 456) 

239 

217 
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Appendix Table C.8 (continued) 


SOURCE: MDRC calculations using data from New York City Department of Education 
administrative records. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the difference between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 
5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. No statistically 
significant differences between subgroup impacts were observed. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. Standard errors were adjusted to account for multiple 
observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Years 1, 2, 3, and 4 cover the 2007-2008, 2008-2009, 2009-2010, and 2010-2011 school years, 
respectively. 

The Regents measures in this table include the following Regents exams: English, Math A, Math 
B, Geometry, Integrated Algebra, Algebra 2/Trigonometry, U.S. History and Government, Global 
History and Geography, Living Environment, Chemistry, Physics, and Earth Science. 

School environment was defined using the average pass rates for the English and math Regents 
exams for all students attending a given school in the two years prior to the study. Schools were then 
divided into thirds based on the ranking (lower, medium, and higher) of the pass rates. 
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Supplementary Tables for Chapter 5 




The Opportunity NYC Demonstration: Family Rewards 
Appendix Table D.l 

Impacts on Parents’ Medicaid Coverage, Years 1 to 3 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Parents' Medicaid coverage 

Ever covered (%) 

Years 1-3 

79.8 

79.9 

-0.2 

0.848 

Year 1 

74.4 

73.7 

0.6 

0.463 

Year 2 

72.6 

71.2 

1.4 

0.171 

Year 3 

71.0 

69.8 

1.1 

0.293 

Always covered (%) 

Years 1-3 

37.3 

34.8 

2.5 ** 

0.041 

Year 1 

54.4 

51.8 

2.6 ** 

0.028 

Year 2 

54.9 

52.0 

29** 

0.018 

Year 3 

52.5 

50.6 

1.9 

0.136 

Average number of covered quarters 

Years 1-3 

7.7 

7.5 

0.2 * 

0.056 

Year 1 

2.6 

2.6 

0.1 * 

0.074 

Year 2 

2.6 

2.5 

0.1 * 

0.072 

Year 3 

2.5 

2.5 

0.1 

0.200 

Sample size (total = 4,995) 

2,515 

2,480 




SOURCE: MDRC calculations from Human Resources Administration (HRA) Medicaid 
coverage data. 

NOTES: A two-tailed t-test was applied to differences between outcomes for the program and 
control groups. The p-value indicates the likelihood that the difference between the program and 
control groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 
percent; ** = 5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 
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The Opportunity NYC Demonstration: Family Rewards 
Appendix Table D.2 


Impacts on Parents’ Medicaid Coverage, 
by Parents’ TANF or SNA Receipt at the Time of Random Assignment 



Program 

Control 

Difference 


Subgroup and Outcome 

Group 

Group 

(Impact) 

P- Value 

Not receiving TANF/SNA at baseline 

Ever covered by Medicaid (%) 





Years 1-3 

74.3 

74.5 

- 0.2 

0.840 

Year 1 

67.8 

67.1 

0.7 

0.539 

Year 2 

67.0 

64.7 

2.3 * 

0.053 

Year 3 

65.3 

63.8 

1.5 

0.242 

Always covered by Medicaid (%) 





Years 1-3 

34.5 

31.1 

3.4 ** 

0.013 

Year 1 

48.8 

45.9 

3.0 ** 

0.025 

Year 2 

50.7 

47.0 

2 7 *** 

0.009 

Year 3 

48.8 

46.3 

2.5 * 

0.087 

Average number of quarters covered by Medicaid 





Years 1-3 

7.1 

6.8 

0.3 ** 

0.030 

Year 1 

2.4 

2.3 

0.1 * 

0.077 

Year 2 

2.4 

2.3 

0.1 ** 

0.023 

Year 3 

2.3 

2.2 

0.1 

0.134 

Sample size (total = 3,716) 

1,848 

1,868 



Receiving TANF/SNA at baseline 

Ever covered by Medicaid (%) 





Years 1-3 

97.6 

96.2 

1.4 

0.105 

Year 1 

96.1 

93.4 

27 ** 

0.026 

Year 2 

91.0 

90.7 

0.3 

0.858 

Year 3 

89.1 

88.0 

1.1 

0.530 

Always covered by Medicaid (%) 





Years 1-3 

45.0 

45.1 

- 0.1 

0.962 

Year 1 

71.7 

70.3 

1.3 

0.612 

Year 2 

67.5 

66.4 

1.1 

0.693 

Year 3 

64.2 

62.7 

1.4 

0.621 

Average number of quarters covered by Medicaid 





Years 1-3 

9.8 

9.7 

0.1 

0.451 

Year 1 

3.4 

3.4 

0.1 

0.242 

Year 2 

3.2 

3.2 

0.0 

0.748 

Year 3 

3.1 

3.1 

0.0 

0.688 

Sample size (total = 1,128) 

583 

545 
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Appendix Table D.2 (continued) 

SOURCE: MDRC calculations from Human Resources Administration (HRA) Medicaid coverage 
data. 

NOTES: A two-tailed t-test was applied to differences between outcomes for the program and 
control groups. The p-value indicates the likelihood that the difference between the program and 
control groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 
percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. No statistically 
significant differences between subgroup impacts were observed. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of families or sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 
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The Opportunity NYC Demonstration: Family Rewards 
Appendix Table D.3 


Impacts on Focal Child’s Health Outcomes, by Child’s School Level 



Program 

Control 

Difference 

Effect 

Outcome 

Group 

Group 

(Impact) 

P- Value Size 

Elementary school students at baseline 





Child's health (1 = poor; 5 = excellent) 

4.0 

4.0 

0.0 

0.781 0.018 

Excellent (%) 

42.4 

39.6 

2.8 

0.383 

Very good (%) 

23.3 

27.8 

-4.5 

0.119 

Good (%) 

25.7 

23.7 

2.0 

0.479 

Fair (%) 

7.7 

7.7 

-0.1 

0.977 

Poor (%) 

0.9 

1.2 

-0.3 

0.655 

Child has any health condition* (%) 

29.9 

29.8 

0.1 

0.971 

Asthma 

11.6 

12.0 

-0.5 

0.831 

Learning disability 

8.0 

8.8 

-0.8 

0.638 

Attention deficit disorder 

3.4 

4.6 

-1.2 

0.337 

Sample size (total = 939) 

486 

453 



Middle school students at baseline 





Child's health (1 = poor; 5 = excellent) 

4.1 

3.9 

0.1 * 

0.051 0.127 

Excellent (%) 

45.9 

38.3 

7.6 ** 

0.021 

Very good (%) 

24.5 

25.8 

-1.3 

0.663 

Good (%) 

21.2 

26.4 

-5.3 * 

0.064 

Fair (%) 

5.9 

8.4 

-2.5 

0.141 

Poor (%) 

2.6 

1.1 

1.5 * 

0.096 

Child has any health condition"* (%) 

26.1 

26.4 

-0.3 

0.914 

Asthma 

9.8 

9.3 

0.5 

0.815 

Learning disability 

5.5 

5.6 

-0.2 

0.918 

Attention deficit disorder 

3.5 

3.9 

-0.4 

0.725 

Sample size (total = 905) 

473 

432 



High school students at baseline 





Child's health (1 = poor; 5 = excellent) 

3.9 

3.9 

0.0 

0.598 0.037 

Excellent (%) 

37.2 

37.0 

0.2 

0.959 

Very good (%) 

27.0 

25.7 

1.3 

0.674 

Good (%) 

26.0 

25.3 

0.7 

0.819 

Fair (%) 

8.5 

10.7 

-2.2 

0.282 

Poor (%) 

1.4 

1.4 

0.0 

0.992 


(continued) 
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Appendix Table D.3 (continued) 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Effect 
P- Value Size 

Child has any health condition* (%) 

24.7 

23.9 

0.8 

0.788 

Asthma 

7.2 

6.3 

0.9 

0.620 

Learning disability 

5.6 

5.5 

0.2 

0.920 

Attention deficit disorder 

2.1 

2.4 

-0.4 

0.732 

Sample size (total = 812) 

429 

383 




SOURCE: MDRC calculations using data from the Family Rewards 42-month survey. 

NOTES: This table presents outcomes only for focal children who were living in the 
household at the time of the survey interview. Over 95 percent of elementary, middle, and high 
school level children were in the 4th, 7th, and 9th target grades at random assignment, 
respectively. 

Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and 
control groups. The p-value indicates the likelihood that the difference between the program 
and control groups arose by chance. Statistical significance levels are indicated as follows: *** 
= 1 percent; ** = 5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre- 
random assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

The effect size is the difference between program and control group outcomes expressed 
as a proportion of the standard deviation of the outcomes for both groups combined. 

a This includes physical, medical, learning, emotional, or mental health conditions. The 
three most commonly reported conditions are listed. Sample members may list multiple 
conditions. 
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The Opportunity NYC Demonstration: Family Rewards 
Appendix Table D.4 


Health Insurance Rewards Earned for Maintaining Coverage, 
by TANF or SNA Receipt at the Time of Random Assignment 





Years 1, 2, and 3 

Outcome 

Year 1 

Year 2 Year 3 a 

Combined 

Families not receiving TANF/SNA at baseline (%) 




Parents 




Earned at least one reward for parents' 




Public heath insurance 

61.4 

63.5 

69.2 

Private health insurance 

25.8 

25.3 

29.9 

Public or private health insurance 

82.4 

84.4 

90.4 

Sample size 



1,848 

Families 




Earned at least one reward for children's 




Public heath insurance 

68.9 

69.1 

74.9 

Private health insurance 

20.0 

18.1 

23.0 

Public or private health insurance 

83.2 

82.6 

89.4 

Sample size 



1,724 

Families receiving TANF/SNA at baseline (%) 




Parents 




Earned at least one reward for parents' 




Public heath insurance 

81.3 

82.5 

89.4 

Private health insurance 

3.1 

4.0 

5.0 

Public or private health insurance 

82.5 

85.1 

90.9 

Sample size 



583 

Families 




Earned at least one reward for children's 




Public heath insurance 

75.4 

77.2 

82.8 

Private health insurance 

2.3 

2.1 

3.0 

Public or private health insurance 

76.3 

78.9 

83.8 

Sample size 



569 


SOURCE: MDRC calculations using Seedco's Family Rewards program data. 

NOTE: The first program year covers September 2007 through August 2008; the second program 
year covers September 2008 through August 2009; and the third program year covers September 
2009 through August 2010. 

a This reward was discontinued in Year 3. 
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Appendix E 

Supplementary Tables for Chapter 6 




The Opportunity NYC Demonstration: Family Rewards 
Appendix Table E.l 

Impacts on Ul-Covered Employment, Earnings, 
and Earnings Categories, Years 1 to 3 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Ever employed (%) 

Years 1-3 

63.3 

64.9 

-1.6 

0.114 

Year 1 

56.3 

58.7 

_2 4 *** 

0.008 

Year 2 

55.0 

56.2 

-1.2 

0.240 

Year 3 

52.5 

53.3 

-0.9 

0.420 

Average quarterly employment (%) 

Years 1-3 

47.7 

48.7 

-1.0 

0.209 

Year 1 

49.1 

50.4 

-1.3 * 

0.098 

Year 2 

47.8 

48.8 

-1.0 

0.273 

Year 3 

46.1 

46.7 

-0.6 

0.512 

Total earnings ($) 

Years 1-3 

36,912 

37,506 

-594 

0.424 

Year 1 

12,154 

12,376 

-221 

0.323 

Year 2 

12,344 

12,601 

-257 

0.369 

Year 3 

12,414 

12,529 

-116 

0.720 

Total earnings, Year 1 (%) 

$1 - $4,999 

9.0 

10.6 

-1.6 ** 

0.047 

$5,000 - $9,999 

6.7 

7.5 

-0.8 

0.282 

$10,000 -$19,999 

14.5 

14.7 

-0.1 

0.886 

$20,000 - $29,999 

11.1 

10.3 

0.8 

0.323 

Over $30,000 

14.9 

15.7 

-0.7 

0.299 

Total earnings, Year 2 (%) 

$1 - $4,999 

9.4 

9.1 

0.3 

0.736 

$5,000 - $9,999 

6.3 

6.6 

-0.3 

0.669 

$10,000 -$19,999 

12.7 

13.9 

-1.1 

0.210 

$20,000 - $29,999 

10.7 

10.4 

0.3 

0.761 

Over $30,000 

16.0 

16.3 

-0.3 

0.682 

Total earnings, Year 3 (%) 

$1 - $4,999 

7.4 

7.8 

-0.4 

0.580 

$5,000 - $9,999 

6.1 

6.0 

0.1 

0.893 

$10,000 -$19,999 

12.1 

13.4 

-1.3 

0.140 

$20,000 - $29,999 

10.3 

10.1 

0.2 

0.826 

Over $30,000 

16.6 

16.0 

0.6 

0.460 

Sample size (total = 4,993) 

2,513 

2,480 




(continued) 
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Appendix Table E.l (continued) 

SOURCE: MDRC calculations using data from New York State unemployment insurance (UI) 
wage records. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and 
control groups. The p-value indicates the likelihood that the difference between the program 
and control groups arose by chance. Statistical significance levels are indicated as follows: *** 
= 1 percent; ** = 5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre- 
random assignment characteristics for sample members. Standard errors were adjusted to 
account for multiple observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Dollar averages include zero values for sample members who were not employed. 

This table includes only employment and earnings in jobs covered by the New York State 
UI program. It does not include employment outside of New York State, nor in jobs not 
covered by the UI system (for example, "off-the -books" jobs and federal government jobs). 
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The Opportunity NYC Demonstration: Family Rewards 
Appendix Table E.2 

Impacts on Ul-Covered Employment and Earnings, by Respondent’s 
Education Level at the Time of Random Assignment 



Program 

Control 

Difference 



Subgroup and Outcome 

Group 

Group 

(Impact) 

P-Value 

Sig. 

High school dk>loma/GED certificate 
or hieher at baseline 






Full sample (UI records) 






Ever employed (%) 






Year 1-3 

71.2 

72.1 

-0.9 

0.451 


Year 1 

65.3 

66.6 

-1.3 

0.261 


Year 2 

63.4 

63.1 

0.3 

0.844 

t 

Year 3 

60.5 

59.9 

0.6 

0.667 


Average quarterly employment (%) 






Year 1-3 

56.3 

55.8 

0.4 

0.682 

tt 

Year 1 

57.9 

58.7 

-0.8 

0.435 


Year 2 

56.5 

55.6 

0.9 

0.442 

tt 

Year 3 

54.3 

53.2 

1.1 

0.395 

tt 

Total earnings ($) 






Year 1-3 

48,320 

48,406 

-86 

0.937 


Year 1 

15,730 

16,084 

-354 

0.276 


Year 2 

16,219 

16,198 

21 

0.960 


Year 3 

16,370 

16,124 

247 

0.611 


Sample size (total = 2,863) 

1,404 

1,459 




No hieh school dmloma/GED certificate 
at baseline 






Full sample (UI records) 






Ever employed (%) 






Years 1-3 

53.0 

55.6 

-2.6 

0.128 


Year 1 

44.2 

48.4 

_4 2 *** 

0.007 


Year 2 

43.9 

47.4 

-3.5 ** 

0.042 

t 

Year 3 

41.8 

44.6 

-2.8 

0.116 


Average quarterly employment (%) 






Years 1-3 

36.2 

39.2 

-3.1 ** 

0.014 

tt 

Year 1 

37.1 

39.6 

-2.5 ** 

0.049 


Year 2 

36.2 

40.0 

-3.8 ** 

0.010 

tt 

Year 3 

35.1 

38.0 

-2.9 * 

0.062 

tt 

(continued) 
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Appendix Table E.2 (continued) 


Subgroup and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value Sig. 

Total earnings ($) 





Years 1-3 

20,730 

22,519 

-1,790 * 

0.060 

Year 1 

7,011 

7,249 

-238 

0.414 

Year 2 

6,875 

7,689 

-815 ** 

0.032 

Year 3 

6,844 

7,580 

-737 * 

0.062 

Sample size (total = 1,960) 

1,021 

939 




SOURCES: MDRC calculations using data from New York State unemployment insurance (Ul) 
wage records. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the differences between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 
5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical 
significance levels (Sig.) are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. Standard errors were adjusted to account for 
multiple observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Dollar averages include zero values for sample members who were not employed. 

This table includes only employment and earnings in jobs covered by the New York State UI 
program. It does not include employment outside of New York State, nor in jobs not covered by the 
UI system (for example, "off-the -books" jobs and federal government jobs). 
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The Opportunity NYC Demonstration: Family Rewards 
Appendix Table E.3 


Impacts on Ul-Covered Employment and Earnings, by Respondent’s 
Employment Status at the Time of Random Assignment 


Subgroup and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) P-Value 

Sig. 

Enmloved at baseline 






Full sample (UI records) 






Ever employed (%) 






Years 1-3 

86.3 

87.2 

-0.8 

0.374 


Year 1 

83.1 

84.2 

-1.1 

0.255 

t 

Year 2 

79.5 

80.3 

-0.9 

0.486 


Year 3 

76.5 

76.4 

0.1 

0.928 


Average quarterly employment (%) 






Years 1-3 

73.8 

73.6 

0.2 

0.853 

t 

Year 1 

77.2 

77.8 

-0.6 

0.533 


Year 2 

73.3 

73.0 

0.4 

0.777 


Year 3 

70.8 

70.0 

0.8 

0.563 


Total earnings ($) 






Years 1-3 

61,296 

61,979 

-683 

0.569 


Year 1 

20,522 

20,837 

-315 

0.372 


Year 2 

20,291 

20,715 

-425 

0.357 


Year 3 

20,484 

20,427 

57 

0.915 


Sample size (total = 2,633) 

1,324 

1,309 




Not enmloved at baseline 






Full sample (UI records) 






Ever employed (%) 






Years 1-3 

37.4 

39.8 

-2.4 

0.192 


Year 1 

25.7 

30.0 

-4.3 ** 

0.011 

t 

Year 2 

27.3 

29.1 

-1.8 

0.295 


Year 3 

25.5 

27.7 

-2.2 

0.201 


Average quarterly employment (%) 






Years 1-3 

18.2 

20.7 

-2.5 ** 

0.033 

t 

Year 1 

17.2 

19.7 

-2.6 ** 

0.042 


Year 2 

19.0 

21.6 

-2.6 * 

0.057 


Year 3 

18.3 

20.7 

-2.4 * 

0.097 



(continued) 
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Appendix Table E.3 (continued) 


Subgroup and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value Sig. 

Total earnings ($) 
Years 1-3 

9,368 

10,007 

-640 

0.447 

Year 1 

2,685 

2,862 

-177 

0.502 

Year 2 

3,384 

3,487 

-103 

0.752 

Year 3 

3,298 

3,659 

-360 

0.293 

Sample size (total = 2,282) 

1,147 

1,135 




SOURCES: MDRC calculations using data from New York State unemployment insurance 
(UI) wage records. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and 
control groups. The p-value indicates the likelihood that the differences between the program 
and control groups arose by chance. Statistical significance levels are indicated as follows: *** 
= 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical 
significance levels (Sig.) are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 
percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre- 
random assignment characteristics of sample members. Standard errors were adjusted to 
account for multiple observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Dollar averages include zero values for sample members who were not employed. 

This table includes only employment and earnings in jobs covered by the New York State 
UI program. It does not include employment outside of New York State, nor in jobs not 
covered by the UI system (for example, "off-the -books" jobs and federal government jobs). 
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The Opportunity NYC Demonstration: Family Rewards 
Appendix Table E.4 


Impacts on Education and Training, by Respondent’s 
Employment Status at the Time of Random Assignment 


Subgroup and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Sig. 

Employed at baseline (%) 






Currently participating in any education 
or training activity 

15.4 

10.9 

4 ^ *** 

0.009 


Has any trade license or training certification 

57.5 

57.1 

0.4 

0.873 

tt 

Has bachelor's degree or higher 

12.6 

11.7 

0.9 

0.568 


Sample size (total = 1,529) 

799 

730 




Not employed at baseline (%) 






Currently participating in any education 
or training activity 

10.5 

9.3 

1.1 

0.489 


Has any trade license or training certification 

50.8 

43.2 

"J -j 

0.004 

tt 

Has bachelor's degree or higher 

7.2 

4.8 

2.4 * 

0.055 


Sample size (total = 1,395) 

721 

674 





SOURCES: MDRC calculations using data from the Family Rewards 42-month survey. 


NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the differences between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 
5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical 
significance levels (Sig.) are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

This table reports on degrees and licenses received, regardless of whether they were received 
before or after random assignment. 
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Opportunity NYC Demonstration: Family Rewards 
Appendix Table E.5 

Impacts on Education and Training, by Respondent’s 
Poverty Level at the Time of Random Assignment 


Subgroup and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Income at or above 50% of FPL 
at baseline (%) 





Currently participating in any education 
or training activity 

15.0 

10.5 

4 ^ *** 

0.005 

Has any trade license or training certification 

56.5 

53.7 

2.8 

0.226 

Has bachelor's degree or higher 

10.5 

10.1 

0.5 

0.742 

Sample size (total = 1,785) 

964 

821 



Income less than 50% of FPL 
at baseline (%) 





Currently participating in any education 
or training activity 

10.4 

9.2 

1.2 

0.494 

Has any trade license or training certification 

51.1 

45.4 

5.7 ** 

0.049 

Has bachelor's degree or higher 

9.1 

5.9 

3 2** 

0.034 

Sample size (total = 1,180) 

579 

601 




SOURCES: MDRC calculations using data from the Family Rewards 42-month survey. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the differences between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. No statistically 
significant differences between subgroup impacts were observed. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

This table reports on degrees and licenses received, regardless of whether they were received 
before or after random assignment. 

FPL = federal poverty level. 
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The Opportunity NYC Demonstration: Family Rewards 
Appendix Table E.6 


Occupations and Industries of Jobs Held by Respondents 
at the Time of the 42-Month Survey 


Outcome 

Survey Data Only 
(Non-UI) 

Ul and Survey 
Data 

Occupation (%) 

Service workers 

61.4 

46.6 

Child care 

35.1 

1.6 

Health care support 

8.3 

21.4 

Clerical workers 

9.1 

16.8 

Maintenance workers 

7.5 

6.0 

Sales-related workers 

3.5 

5.0 

Management, business/fmance 

4.0 

5.0 

Teaching assistants and school aides 

3.5 

9.5 

Other 

11.0 

11.2 

Industry (%) 

Transportation, warehousing, and utilities 
Construction, manufacturing, wholesale trade, 

3.0 

4.3 

and retail trade 

7.4 

8.0 

Finance, insurance, real estate, rental, and leasing 

1.1 

2.6 

Educational services 

3.6 

11.7 

Health care and social services 

50.0 

43.3 

Food preparation and accomondation 

3.0 

3.1 

Administrative, support, and waste management 
Public administration, information, arts and 

6.8 

8.7 

entertainment, professional, and other services 

14.2 

8.5 

Other 

10.9 

9.9 

Sample size (total = 1,562) 

373 

1,189 


SOURCES: MDRC calculations using data from the Family Rewards 42-month survey and New 
York State unemployment insurance (Ul) administrative records. 

NOTES: Sample sizes may vary because of missing values. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Estimates include only the sample members who reported that they were employed at the time 
of the survey interview. 
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The Opportunity NYC Demonstration: Family Rewards 
Appendix Table E.7 

Impacts on Employment, Earnings, and Average Rewards Earned, 
by Respondent’s Employment Status at the Time of Random Assignment 

Program Control Difference 

Subgroup and Outcome Group Group (Impact) P-Value Sig. 


Employed at baseline 
Full sample, UI records 

Average quarterly employment, Years 1-3 (%) 
Total earnings, Years 1-3 ($) 

Sample size (total = 2,633) 

Survey sample 
Self-reported responses 
Currently working (%) 

Ever worked since random assignment (%) 
Number of months worked in past year 
UI records 

Working in quarter of survey interview, 
according to UI (%) 

Average quarterly employment. Years 1-3 (%) 
Total earnings, Years 1-3 ($) 

Sample size (total = 1,529) 

Rewards earned, program group families 
Amount of education rewards earned ($) 

Amount of health rewards earned ($) 

Amount of workforce rewards earned ($) 

Total rewards earned ($) 

Sample size 


73.8 

73.6 

0.2 

0.853 t 

61,296 

61,979 

-683 

0.569 

1,324 

1,309 




82.3 

75.2 

7 Q *** 

0.001 

96.3 

96.0 

0.3 

0.786 

9.7 

9.0 

0 7 *** 

0.003 


72.2 

68.5 

3.7 * 

0.074 ff 

75.9 

75.4 

0.5 

0.717 

62,550 

61,554 

995 

0.513 

799 

730 



4,094 

— 

— 


3,099 


- 


2,903 


- 


10,095 

— 

— 



1,235 


(continued) 
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Appendix Table E.7 (continued) 


Subgroup and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Sig. 

Not enmloved at baseline 
Full sample, UI records 

Average quarterly employment, Years 1-3 (%) 

18.2 

20.7 

-2.5 ** 

0.033 

t 

Total earnings, Years 1-3 ($) 

9,368 

10,007 

-640 

0.447 


Sample size (total = 2,282) 
Survey sample 
Self-reported responses 

1,147 

1,135 




Currently working (%) 

27.5 

23.1 

4.4 * 

0.054 


Ever worked since random assignment (%) 

57.2 

55.1 

2.2 

0.387 


Number of months worked in past year 

3.5 

3.1 

0.4 * 

0.097 


UI records 






Working in quarter of survey interview, 
according to UI (%) 

18.9 

21.1 

-2.2 

0.294 

tt 

Average quarterly employment. Years 1-3 (%) 

18.1 

20.4 

-2.3 

0.129 


Total earnings, Years 1-3 ($) 

8,948 

9,405 

-457 

0.661 


Sample size (total = 1,395) 

721 

674 




Rewards earned, program group families 

Amount of education rewards earned ($) 

3,728 





Amount of health rewards earned ($) 

2,876 

- 

- 



Amount of workforce rewards earned ($) 

599 

- 

- 



Total rewards earned ($) 

7,203 

- 

- 



Sample size 

1,103 






(continued) 
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Appendix Table E.7 (continued) 

SOURCES: MDRC calculations using data from the Family Rewards 42-month survey and New York 
State unemployment insurance (UI) wage records, and Seedco's Family Rewards program data. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the differences between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical significance 
levels (Sig.) are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. Standard errors were adjusted to account for multiple 
observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Dollar averages include zero values for sample members who were not employed. 

UI records include only employment and earnings in jobs covered by the New York State UI 
program. They do not include employment outside of New York State, nor in jobs not covered by the 
UI system (for example, "off-the-books" jobs and federal government jobs). 

A double dash (--) indicates "not applicable." 
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Opportunity NYC Demonstration: Family Rewards 
Appendix Table E.8 

Impacts on Employment, Earnings, and Average Rewards Earned, 
by Respondent’s Poverty Level at the Time of Random Assignment 


Subgroup and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value Sig. 

Income at or above 50% of FPL 
at baseline 





Full sample, UI records 

Average quarterly employment, Years 1-3 (%) 

66.2 

66.6 

-0.3 

0.725 f 

Total earnings, Years 1-3 ($) 

55,026 

55,435 

-409 

0.708 

Sample size (total = 3,062) 
Survey sample 
Self-reported responses 

1,584 

1,478 



Currently working (%) 

67.9 

62.0 

5 g *h=h= 

0.002 

Ever worked since random assignment (%) 

87.4 

86.0 

1.4 

0.344 

Number of months worked in past year 

8.1 

7.5 

0.6 *** 

0.007 

Ul records 





Working in quarter of survey interview, 
according to UI (%) 

63.7 

62.0 

1.6 

0.392 

Average quarterly employment, Years 1-3 (%) 

67.6 

67.7 

-0.1 

0.956 

Total earnings, Years 1-3 ($) 

55,558 

54,561 

997 

0.472 

Sample size (total = 1,785) 

964 

821 



Rewards earned, program group families 





Amount of education rewards earned ($) 

3,998 

- 

- 


Amount of health rewards earned ($) 

3,076 

- 

- 


Amount of workforce rewards earned ($) 

2,395 

- 

- 


Total rewards earned ($) 

9,469 

- 

- 


Sample size 

1,478 
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Appendix Table E.8 (continued) 



Program 

Control 

Difference 


Subgroup and Outcome 

Group 

Group 

(Impact) 

P- Value Sig. 

Income less than 50% of FPL at baseline 





Full sample, UI records 





Average quarterly employment, Years 1-3 (%) 

17.6 

20.8 

_3 2 ** 

0.011 t 

Total earnings. Years 1-3 ($) 

7,946 

9,281 

-1,335 * 

0.095 

Sample size (total = 1,931) 
Survey sample 
Self-reported responses 

929 

1,002 



Currently working (%) 

38.0 

30.9 

2 2 *** 

0.004 

Ever worked since random assignment (%) 

63.2 

60.2 

3.0 

0.225 

Number of months worked in past year 

4.6 

4.0 

0.6 ** 

0.015 

UI records 





Working in quarter of survey interview, 
according to UI (%) 

20.4 

20.2 

0.2 

0.928 

Average quarterly employment. Years 1-3 (%) 

18.0 

20.2 

-2.2 

0.176 

Total earnings, Years 1-3 ($) 

7,815 

8,807 

-992 

0.312 

Sample size (total = 1,180) 

579 

601 



Rewards earned, program group families 





Amount of education rewards earned ($) 

3,736 

- 

- 


Amount of health rewards earned ($) 

2,818 

- 

- 


Amount of workforce rewards earned ($) 

813 

- 

- 


Total rewards earned ($) 

7,368 

- 

- 


Sample size 

899 
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Appendix Table E.8 (continued) 

SOURCES: MDRC calculations using data from the Family Rewards 42-month survey and New York 
State unemployment insurance (Ul) wage records, and Seedco's Family Rewards program data. 

NOTES: Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to differences between outcomes for the program and control 
groups. The p-value indicates the likelihood that the differences between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical significance 
levels (Sig.) are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. Standard errors were adjusted to account for multiple 
observations per family. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Dollar averages include zero values for sample members who were not employed. 

UI records include only employment and earnings in jobs covered by the New York State UI 
program. They do not include employment outside of New York State, nor in jobs not covered by the 
Ul system (for example, "off-the-books" jobs and federal government jobs). 

A double dash (— ) indicates "not applicable." 

FPL = federal poverty level. 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to leam not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school reforms, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 



